Ash's TensorFold 1.2.7 on one RTX 5090: Qwen3.8-27B (NVIDIA NVFP4, DFlash2 drafts), window 131,072 tokens

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds; long prompts (later supplementary run of 2026-11-11, same setup): prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests. Measured 2026-11-11. Raw files: speed result · speed contract · log · long answer · speed result · speed contract · log.

What ran. The speed test ran TensorFold 1.2.7 as the official release archive for x86_64 Linux (tensorfold-1.2.7-linux-x86_64.tar.gz, published 11 Nov 2026, 17:18 UTC by the repository owner, sha256 319f548b, the digest GitHub lists for that file; the binary tensorfold-native inside has the sha256 8e5729ba that the measuring machine recorded before the runs; no patch, no container) and the recipe's command of docs/recipes/qwen3.8-27b.md lines 249 and 250 (tensorfold pull nvidia/Qwen3.8-27B-NVFP4 z-lab/Qwen3.8-27B-DFlash2, then tensorfold serve nvidia/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8080), started with the binary tensorfold-native of that archive (README line 79 defines tensorfold as a call of tensorfold-native). The checkpoint is nvidia/Qwen3.8-27B-NVFP4 at revision 482ca0f3 and the drafter is z-lab/Qwen3.8-27B-DFlash2 at revision 50307d4c; the marker line of the file check is in the start text recorded with the run. I started it with --drafter z-lab/Qwen3.8-27B-DFlash2 (README line 323: Qwen3.8-27B uses DFlash2 through that flag), a name, and a local host and port and, for this page, --context 131072 (see Window and streams). Hardware of the docs. The recipe's NVFP4 section (line 256 onward) gives its decode numbers for one DGX Spark (GB10), and its CUDA table lists bf16 prompts against --prefill-fp8 as 'not measured yet' (line 229). The README says that the CUDA release images include sm_80, sm_86, sm_120 and sm_121 (line 170). This page is a measurement on one RTX 5090; the docs' numbers are not predictions for this card, and I show none of them here. Window and streams. The window is 131,072 tokens, set with --context 131072; the server's default is 32,768 tokens (README line 307) and the model's own window is 262,144 tokens. My tests need more than 32,768 tokens in one window: LiveCodeBench answers of up to 32,768 tokens, the 32K long-text prompts (32,779 and 32,780 tokens) and the long-prompt steps from 32K. The README says that --context bounds prompt plus reply (line 307). This is my deviation from the command, and every number on this page was measured with it. With this window the server chose the streams itself (--parallel auto asks for up to eight streams as memory and the engine allow, README lines 104 and 345): its start line reads '1 stream at once, 8.96 GiB each at a 131072-token window, of 9.1 GiB left after a 1.0 GiB reserve' (the model takes 20.78 GiB with the drafter), and GET /v1/models reports a context of 131072 (start line and speed result, recorded with the run). Long prompts. The 128K step could not use 131,072 prompt tokens: with 512 reply tokens that is 131,584 tokens against the window of 131,072. The test then takes the window minus 1,024, which is 130,048 tokens, as its rule says (valid from 126,976). The row for 128K in the table of long prompts was therefore measured at 130,048 tokens; the other three steps (16K, 32K, 64K) have the exact length. The prefill series of this card has two sizes, 2K and 8K, so the tiles for prefill and first token stop at 8K; the values for 16K, 32K, 64K and 128K are in the table of long prompts. The long-prompt steps were measured twice. The speed run itself contained them (raw folder ash27nvfp4-5090-127-ctx131072-1111-speed-r0), and a supplementary run of the same day repeated only these steps (raw folder ash27nvfp4-5090-127-ctx131072-1111-ctx-speed-r0-ctx, same command, the server started anew). The table shows the supplementary run. The values of the speed run are: 16K: 192.8 tok/s (code) and 182.4 tok/s (prose), prefill 3445.1 tok/s, first token after 4.8 s; 32K: 196.7 tok/s (code) and 177.8 tok/s (prose), prefill 3224.4 tok/s, first token after 10.2 s; 64K: 179.8 tok/s (code) and 163.7 tok/s (prose), prefill 2871.4 tok/s, first token after 22.8 s; 128K step (130,048 tokens): 155.6 tok/s (code) and 135.0 tok/s (prose), prefill 2385.9 tok/s, first token after 54.5 s. Requests at once. The server has 1 stream. '1 at once' is inside it; '2', '4' and '8 at once' send 2, 4 and 8 requests to 1 stream (the others wait), and the tool logs them as measured anyway. So these rows measure the waiting requests as well: '8 at once' does not mean 8 streams, and the rows for 2, 4 and 8 at once are not comparable with the row for 1 at once. The total falls from 181.0 tok/s with 1 request to 169.6 tok/s with 8 requests, while the speed of one request falls from 181.0 to 37.4 tok/s. The headline 214.7 tok/s is the mean of the code prompt (224.7) and the prose prompt (204.8), medians of two runs each (spread (CV) of 2.9 % and 0.3 %). The code prompt has 1,258 and 1,260 tokens and the prose prompt 1,352 and 1,355 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 19.2 % (1,520 of 7,905 drafted tokens, mean accepted length 3.8 tokens). The long-answer test asks for one reply of up to 32,768 tokens; the reply had 5,793 tokens and ended with a stop after 20.4 seconds. Sampling is the Qwen model card's setting for replies without thinking (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5), sent with the request flag for no thinking and seed 42; the contracts of the speed runs name the source. The chat template thinks by default and the command passes no sampling flag, so the server's own default is temperature 1.0, top_p 0.95, top_k 20 (start line). Foreign load. The load guard (any other process above 25 % of a core for two samples in a row, sampled twice a second) was set to ignore one kernel thread, kcompactd0 (memory compaction). In all 35 steps of the speed run and all 16 steps of the supplementary run it found no other process above its limit, so no step was repeated; the highest other process in any step was the Docker daemon (dockerd) at 18.0 % of a core. The quiet check of the speed run (one sample about every 10 seconds) took 57 samples and the one of the supplementary run 43 samples; all of them show only kernel threads (kswapd0 and one kworker), at most 2.2 % of a core.

python3 speed2.py --base-url http://127.0.0.1:41365/v1 --label ash27nvfp4-5090-127-ctx131072-1111-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/ash27nvfp4-5090-127-ctx131072-1111-speed-r0/ash27nvfp4-5090-127-ctx131072-1111-speed-r0/speed2 --contract $RUN/ash27nvfp4-5090-127-ctx131072-1111-speed-r0/ash27nvfp4-5090-127-ctx131072-1111-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/ash27b-nvfp4-1111/model/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253; the engine default of the author's command is the checkpoint's generation_config.json (server line: sampling temperature 1.0, top_k 20, top_p 0.95) --model qwen38-27b-nvfp4 --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source TensorFold v1.2.7 README.md:104 and :345 (--parallel auto, the default the recipe command uses, asks for up to eight streams as memory and the engine allow); streams = the server's own choice, read from a startup line of this setup, Nov 11, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 20.78 GiB; 1 stream at once, 8.96 GiB each at a 131072-token window, of 9.1 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' --server-ctx 131072 --server-ctx-source DEVIATION named: --context 131072 (see _comment). Startup line of this setup, Nov 11, 2026 (release 1.2.7, RTX 5090): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 20.78 GiB; 1 stream at once, 8.96 GiB each at a 131072-token window, of 9.1 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' and '[tensorfold] serving qwen38-27b-nvfp4 at http://127.0.0.1:8132/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; drafts: DFlash2 (z-lab/Qwen3.8-27B-DFlash2) + copies; context: 131072; loaded in 14.5s)'. Window source: README.md:307 (v1.2.7), row of --context N: “Bound prompt plus reply tokens; the model's window and memory checks still apply. Qwen3.8-27B defaults to 32,768 and takes up to 262,144.”; the recipe (docs/recipes/qwen3.8-27b.md:248-251) names no --context.

# Long prompts: supplementary run of 2026-11-11 (raw folder ash27nvfp4-5090-127-ctx131072-1111-ctx-speed-r0-ctx)
python3 speed2.py --base-url http://127.0.0.1:33759/v1 --label ash27nvfp4-5090-127-ctx131072-1111-ctx-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/ash27nvfp4-5090-127-ctx131072-1111-ctx-speed-r0/ash27nvfp4-5090-127-ctx131072-1111-ctx-speed-r0/speed2 --contract $RUN/ash27nvfp4-5090-127-ctx131072-1111-ctx-speed-r0/ash27nvfp4-5090-127-ctx131072-1111-ctx-speed-r0/speed2/speed-contract.json --parts ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/ash27b-nvfp4-1111/model/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253; the engine default of the author's command is the checkpoint's generation_config.json (server line: sampling temperature 1.0, top_k 20, top_p 0.95) --model qwen38-27b-nvfp4 --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source TensorFold v1.2.7 README.md:104 and :345 (--parallel auto, the default the recipe command uses, asks for up to eight streams as memory and the engine allow); streams = the server's own choice, read from a startup line of this setup, Nov 11, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 20.78 GiB; 1 stream at once, 8.96 GiB each at a 131072-token window, of 9.1 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' --server-ctx 131072 --server-ctx-source DEVIATION named: --context 131072 (see _comment). Startup line of this setup, Nov 11, 2026 (release 1.2.7, RTX 5090): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 20.78 GiB; 1 stream at once, 8.96 GiB each at a 131072-token window, of 9.1 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' and '[tensorfold] serving qwen38-27b-nvfp4 at http://127.0.0.1:8132/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; drafts: DFlash2 (z-lab/Qwen3.8-27B-DFlash2) + copies; context: 131072; loaded in 14.5s)'. Window source: README.md:307 (v1.2.7), row of --context N: “Bound prompt plus reply tokens; the model's window and memory checks still apply. Qwen3.8-27B defaults to 32,768 and takes up to 262,144.”; the recipe (docs/recipes/qwen3.8-27b.md:248-251) names no --context. --supplement-of ash27nvfp4-5090-127-ctx131072-1111

How well it answers

Three of the seven Quality v2 tests were run with the window of 131,072 tokens and 1 request at a time, the 1 stream of the server: LiveCodeBench and long text at 8K and 32K. Thinking was off in all of them. The other four tests (MMLU-Pro, GPQA Diamond, IFEval and the thinking test) and the memory test are on the page of the same setup with the server's default window.

Thinking off in all three tests. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-11.

Note from the run: Three of the seven Quality v2 tests were run with the window of 131,072 tokens: LiveCodeBench and long text at 8K and 32K. LiveCodeBench ran with an answer limit of 32,768 tokens, long text with 128, both with thinking off and 1 request at a time (the server has 1 stream). A cut-off answer counts as wrong. The other four tests (MMLU-Pro, GPQA Diamond, IFEval and the thinking test) are on the page of the same setup with the server's default window.

LiveCodeBench: the 40 newest tasks of the set (contest dates 2025-03-15 to 2025-04-06), pass@1, the code runs in a sandbox without network. 23 of 40 passed. Of the 17 that failed, 3 were cut off at 32,768 tokens and count as wrong and 2 had no code. By difficulty, 7 of 9 easy, 11 of 11 medium and 5 of 20 hard tasks passed (counted from the answer file, which the site does not publish).

Long text at 8K and 32K, 4 tasks of 10 each, scored strictly: multi-key retrieval 10 of 10 and 10 of 10, variable tracking 7 of 10 and 5 of 10, common-word extraction 10 of 10 and 9 of 10, multi-hop question answering 8 of 10 and 3 of 10. A variable-tracking task counts only when all variables are named (mean fraction 0.825 at 8K and 0.625 at 32K), and a multi-hop answer counts when the expected answer, normalised (lower case, no punctuation or articles), appears in the reply as a run of words. The 32K prompts have 32,779 and 32,780 tokens. Three of the 80 prompts are one token away from the exact length (8,191, 8,193 and 32,767 tokens; the tool marks them not exact).

Sampling. Both tests sent the Qwen model card's setting for replies without thinking (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5) with the request flag for no thinking; every request sent cache_prompt false and seed 42. One run of each test, no repetition.

Foreign load. These tests have no load guard; their quiet check lists the busiest other process about every 10 seconds. LiveCodeBench and long text: 156 samples, 4 of them above 25 % of a core (99.1, 99.0, 96.2, 40.0 %: 3 of them the code sandbox of the LiveCodeBench test itself, a Python process running the solution under test, and 1 a Python one-liner that opens a local web address (I did not trace which program starts it)). The other 152 samples are below 25 % of a core, at most 21.4 %. Nothing was repeated. The card was empty (2 MiB in use) before the server started. Not published: the LiveCodeBench answer file and its code and test folders (the site's filter holds back source files), the quiet-check tables, and the prompt files of the long-text tests; the summaries, logs and contracts of both tests and the answer file of the long-text test are published.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-nvfp4-dflash2-tensorfold-1.2.7-ash-recipe-5090-ctx131072-1111.json --parts lcb --job ash27nvfp4-5090-127-ctx131072-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-nvfp4-dflash2-tensorfold-1.2.7-ash-recipe-5090-ctx131072-1111.json --parts ruler --job ash27nvfp4-5090-127-ctx131072-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-nvfp4-dflash2-tensorfold-1.2.7-ash-recipe-5090-ctx131072-1111.json --parts ruler --job ash27nvfp4-5090-127-ctx131072-1111

Also made this possible

• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• Quant (NVFP4, NVIDIA Model Optimizer, revision 482ca0f3)NVIDIAHugging Face GitHub Website
• Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)Inco AI@inco_ai Hugging Face GitHub Website
• Speculative decoding method (DFlash) and the mirror we loadedz-lab@zhijianliu_ GitHub Hugging Face

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one GPU, the whole model in video memory). Host: AMD Ryzen 9 9950X, 96 GB RAM installed (92 GiB usable). Driver 610.43.02, kernel 6.12.95+deb13-amd64

Model

nvidia/Qwen3.8-27B-NVFP4 @482ca0f3 (ModelOpt NVFP4 MLP and head, FP8 attention and DeltaNet projections, bf16 embedding and gates) + z-lab/Qwen3.8-27B-DFlash2 @50307d4c (drafter)

Quant (NVFP4, NVIDIA Model Optimizer, revision 482ca0f3)

huggingface.co/nvidia/Qwen3.8-27B-NVFP4

Draft model (DFlash2, revision 50307d4c)

huggingface.co/z-lab/Qwen3.8-27B-DFlash2

Engine

TensorFold 1.2.7, official release for x86_64 Linux (native CUDA, sm_120), NVIDIA's NVFP4 checkpoint with DFlash2 drafts, window 131,072 tokens, 1 stream chosen by the server — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.7

Context window

131,072 tokens: how much text the model can see at once

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8132, only from this computer.

   Copy

   tensorfold-native serve nvidia/Qwen3.8-27B-NVFP4 \
     --drafter z-lab/Qwen3.8-27B-DFlash2 \
     --name qwen38-27b-nvfp4 \
     --host 127.0.0.1 \
     --port 8132 \
     --context 131072

   What the parts mean

   • --host 127.0.0.1who can connect: only this computer
   • --port 8132the network port the server listens on
   • --drafter z-lab/Qwen3.8-27B-DFlash2the draft model: it guesses the next tokens and the main model checks them; without it the model decodes plain
   • --name qwen38-27b-nvfp4the model ID the server advertises to clients

TensorFold native (Zig) 1.2.7 official release archive tensorfold-1.2.7-linux-x86_64.tar.gz (sha256 319f548b), CUDA (chip nvidia-sm120), DFlash2 drafts, window 131072 (named deviation --context 131072), 1x RTX 5090 (sm_120); tensorfold-native serve nvidia/Qwen3.8-27B-NVFP4 --drafter z-lab/Qwen3.8-27B-DFlash2 --name qwen38-27b-nvfp4 --host 127.0.0.1 --port 8132 --context 131072 (recipe: tensorfold pull nvidia/Qwen3.8-27B-NVFP4 z-lab/Qwen3.8-27B-DFlash2; tensorfold serve nvidia/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8080, docs/recipes/qwen3.8-27b.md:248-251; I started it with --drafter z-lab/Qwen3.8-27B-DFlash2 (README.md:323), a name, a local host and port and --context 131072 (the default is 32,768, README.md:307))