Ash's TensorFold 1.2.7 on one RTX 5090: Qwen3.8-27B (NVIDIA NVFP4, DFlash2 drafts), window 32,768 tokens

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds; long prompts (later supplementary run of 2026-11-11, same setup): prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests. Measured 2026-11-11. Raw files: speed result · speed contract · log · long answer (empty file, no valid result) · speed result · speed contract · log.

What ran. The speed test ran TensorFold 1.2.7 as the official release archive for x86_64 Linux (tensorfold-1.2.7-linux-x86_64.tar.gz, published 11 Nov 2026, 17:18 UTC by the repository owner, sha256 319f548b, the digest GitHub lists for that file; the binary tensorfold-native inside has the sha256 8e5729ba that the measuring machine recorded before the runs; no patch, no container) and the recipe's command of docs/recipes/qwen3.8-27b.md lines 249 and 250 (tensorfold pull nvidia/Qwen3.8-27B-NVFP4 z-lab/Qwen3.8-27B-DFlash2, then tensorfold serve nvidia/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8080), started with the binary tensorfold-native of that archive (README line 79 defines tensorfold as a call of tensorfold-native). The checkpoint is nvidia/Qwen3.8-27B-NVFP4 at revision 482ca0f3 and the drafter is z-lab/Qwen3.8-27B-DFlash2 at revision 50307d4c; the marker line of the file check is in the start text recorded with the run. I started it with --drafter z-lab/Qwen3.8-27B-DFlash2 (README line 323: Qwen3.8-27B uses DFlash2 through that flag), a name, and a local host and port. Hardware of the docs. The recipe's NVFP4 section (line 256 onward) gives its decode numbers for one DGX Spark (GB10), and its CUDA table lists bf16 prompts against --prefill-fp8 as 'not measured yet' (line 229). The README says that the CUDA release images include sm_80, sm_86, sm_120 and sm_121 (line 170). This page is a measurement on one RTX 5090; the docs' numbers are not predictions for this card, and I show none of them here. Window and streams. The window is the server's default of 32,768 tokens (README line 307). The server chose the streams itself (--parallel auto asks for up to eight streams as memory and the engine allow, README lines 104 and 345): its start line reads '3 streams at once, 2.96 GiB each at a 32768-token window, of 9.1 GiB left after a 1.0 GiB reserve' (the model takes 20.78 GiB with the drafter), and GET /v1/models reports a context of 32768 (start line and speed result, recorded with the run). A prompt and its reply must fit in the window: 32,768 prompt tokens plus 512 reply tokens do not fit, so of the long-prompt series only the 16K step has a row; the steps 32K, 64K and 128K do not fit and are not in the table. The page of the same setup with the larger window measures them. The long-answer test (one reply of up to 32,768 tokens, the same size as the window) has no valid result in this setup: the server returned no output for it (the log of the speed run reads 'L1 done: None tokens in 0.0s, finish_reason None'), the tool marked the test invalid ('stream emitted no output', field l1.invalid_reason of the speed result), and the published file l1-output.txt of the long answer is empty. The words and characters per second come from the four decode runs. Long prompts. The table shows the supplementary run of the same day (raw folder ash27nvfp4-5090-127-1111-ctx-speed-r0-ctx; same command, the server started anew, 4 guarded steps). The speed run itself contained the 16K step too (raw folder ash27nvfp4-5090-127-1111-speed-r0); its values are: 16K: 186.1 tok/s (code) and 174.7 tok/s (prose), prefill 3449.2 tok/s, first token after 4.8 s. The prefill series of this card has two sizes, 2K and 8K, so the tiles for prefill and first token stop at 8K. Requests at once. The server has 3 streams. The levels 1 and 2 are inside them. '4 at once' sends 4 requests to 3 streams (1 waits) and '8 at once' sends 8 requests to 3 streams (5 wait); the tool logs both as measured anyway. So the rows for 4 and 8 at once measure the waiting requests as well: '8 at once' does not mean 8 streams, and these rows are not comparable with the rows inside the streams. The total rises from 193.4 tok/s with 1 request to 282.2 tok/s with 8 requests, while the speed of one request falls from 193.4 to 48.6 tok/s. The headline 214.8 tok/s is the mean of the code prompt (236.3) and the prose prompt (193.2), medians of two runs each (spread (CV) of 15.8 % and 19.4 %). The code prompt has 1,255 and 1,260 tokens and the prose prompt 1,351 and 1,353 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 18.7 % (1,506 of 8,060 drafted tokens, mean accepted length 3.7 tokens). Sampling is the Qwen model card's setting for replies without thinking (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5), sent with the request flag for no thinking and seed 42; the contracts of the speed runs name the source. The chat template thinks by default and the command passes no sampling flag, so the server's own default is temperature 1.0, top_p 0.95, top_k 20 (start line). Foreign load. The load guard (any other process above 25 % of a core for two samples in a row, sampled twice a second) was set to ignore one kernel thread, kcompactd0 (memory compaction). In all 23 steps of the speed run and all 4 steps of the supplementary run it found no other process above its limit, so no step was repeated; the highest other process in any step was the Docker daemon (dockerd) at 16.0 % of a core. The quiet check of the speed run (one sample about every 10 seconds) took 12 samples and the one of the supplementary run 4 samples; all of them show only the kernel's memory thread kswapd0, at most 1.7 % of a core.

python3 speed2.py --base-url http://127.0.0.1:39987/v1 --label ash27nvfp4-5090-127-1111-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/ash27nvfp4-5090-127-1111-speed-r0/ash27nvfp4-5090-127-1111-speed-r0/speed2 --contract $RUN/ash27nvfp4-5090-127-1111-speed-r0/ash27nvfp4-5090-127-1111-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/ash27b-nvfp4-1111/model/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253; the engine default of the author's command is the checkpoint's generation_config.json (server line: sampling temperature 1.0, top_k 20, top_p 0.95) --model qwen38-27b-nvfp4 --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 3 --server-slots-source TensorFold v1.2.7 README.md:104 and :345 (--parallel auto, the default the recipe command uses, asks for up to eight streams as memory and the engine allow); streams = the server's own choice, read from the startup line at the probe start Nov 11, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 20.78 GiB; 3 streams at once, 2.96 GiB each at a 32768-token window, of 9.1 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' --server-ctx 32768 --server-ctx-source The author's command sets no context: the server default window. Startup line of the probe start Nov 11, 2026 (release 1.2.7, RTX 5090): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 20.78 GiB; 3 streams at once, 2.96 GiB each at a 32768-token window, of 9.1 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' and '[tensorfold] serving qwen38-27b-nvfp4 at http://127.0.0.1:8132/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; drafts: DFlash2 (z-lab/Qwen3.8-27B-DFlash2) + copies; context: 32768; loaded in 23.2s)'. Window source: README.md:307 (v1.2.7), row of --context N: “Bound prompt plus reply tokens; the model's window and memory checks still apply. Qwen3.8-27B defaults to 32,768 and takes up to 262,144.”; the recipe (docs/recipes/qwen3.8-27b.md:248-251) names no --context.

# Long prompts: supplementary run of 2026-11-11 (raw folder ash27nvfp4-5090-127-1111-ctx-speed-r0-ctx)
python3 speed2.py --base-url http://127.0.0.1:36391/v1 --label ash27nvfp4-5090-127-1111-ctx-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/ash27nvfp4-5090-127-1111-ctx-speed-r0/ash27nvfp4-5090-127-1111-ctx-speed-r0/speed2 --contract $RUN/ash27nvfp4-5090-127-1111-ctx-speed-r0/ash27nvfp4-5090-127-1111-ctx-speed-r0/speed2/speed-contract.json --parts ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/ash27b-nvfp4-1111/model/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253; the engine default of the author's command is the checkpoint's generation_config.json (server line: sampling temperature 1.0, top_k 20, top_p 0.95) --model qwen38-27b-nvfp4 --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 3 --server-slots-source TensorFold v1.2.7 README.md:104 and :345 (--parallel auto, the default the recipe command uses, asks for up to eight streams as memory and the engine allow); streams = the server's own choice, read from the startup line at the probe start Nov 11, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 20.78 GiB; 3 streams at once, 2.96 GiB each at a 32768-token window, of 9.1 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' --server-ctx 32768 --server-ctx-source The author's command sets no context: the server default window. Startup line of the probe start Nov 11, 2026 (release 1.2.7, RTX 5090): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 20.78 GiB; 3 streams at once, 2.96 GiB each at a 32768-token window, of 9.1 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' and '[tensorfold] serving qwen38-27b-nvfp4 at http://127.0.0.1:8132/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; drafts: DFlash2 (z-lab/Qwen3.8-27B-DFlash2) + copies; context: 32768; loaded in 23.2s)'. Window source: README.md:307 (v1.2.7), row of --context N: “Bound prompt plus reply tokens; the model's window and memory checks still apply. Qwen3.8-27B defaults to 32,768 and takes up to 262,144.”; the recipe (docs/recipes/qwen3.8-27b.md:248-251) names no --context. --supplement-of ash27nvfp4-5090-127-1111

How well it answers

Five of the seven Quality v2 tests were run with the server's default window of 32,768 tokens and 3 requests at a time, the 3 streams of the server: MMLU-Pro, GPQA Diamond, IFEval, long text at 8K and the thinking test (GPQA 30). Thinking was off in four of them; GPQA 30 ran with thinking on. LiveCodeBench and long text at 32K need a larger window and are on the page of the same setup with the larger window.

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-11.

Note from the run: MMLU-Pro and GPQA Diamond ran with an answer limit of 48 tokens (the tool's default; the questions, the seed 42 and the first-letter extraction are the usual ones), IFEval with 4,096 tokens (prompt-level strict), long text with 128 and GPQA 30 with thinking with 16,384. Cut off at their limits: 0 of 400 MMLU-Pro, 0 of 198 GPQA and 0 of 200 IFEval answers and none in the long-text tests; 8 of the 30 thinking answers were cut off at the limit. A cut-off answer counts as wrong. The questions of each test ran 3 at a time, the 3 streams of the server (same questions, limits and seed for every question), so the median time per question of the thinking test was measured under that load: it is not a speed value. GPQA Diamond ran from the public mirror fingertap/GPQA-Diamond (198 questions), because the original is gated and no Hugging Face token was available.

Long text at 8K, 4 tasks of 10 each, scored strictly: multi-key retrieval 10 of 10, variable tracking 7 of 10, common-word extraction 10 of 10, multi-hop question answering 8 of 10. A variable-tracking task counts only when all variables are named, and a multi-hop answer counts when the expected answer, normalised (lower case, no punctuation or articles), appears in the reply as a run of words. Two of the 40 prompts are one token away from the exact length (8,191 and 8,193 tokens; the tool marks them not exact).

Long text at 32K and LiveCodeBench have no result here. The long-text test also ran at 32K, but all 40 of its requests were refused with HTTP 400: the prompts (32,779 or 32,780 tokens) do not fit a window of 32,768 tokens. That is no result of the model, so it is neither in the table nor in the register; the answer file ruler.jsonl lists the refusals. LiveCodeBench was not run in this setup, because its answers can be as long as the window. Both tests are on the page of the same setup with the larger window.

Thinking test (GPQA 30, thinking on): 30 of the GPQA questions, with the thinking-mode setting of the Qwen model card (temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence penalty 0; the contract names the source), seed 42, a limit of 16,384 tokens. 8 of the 30 were still thinking at the limit and count as wrong; of the 22 answers that finished, 21 are right. 70.0 % is the score at this limit. The median of 7,865 tokens and 52.5 seconds per question covers all 30 questions, including the 8 at the limit.

Sampling. The tests with thinking off sent the Qwen model card's setting for replies without thinking (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5) with the request flag for no thinking; every request of the quality tests sent cache_prompt false and seed 42. One run of each test, no repetition.

Foreign load. These tests have no load guard; their quiet check lists the busiest other process about every 10 seconds. MMLU-Pro, GPQA Diamond, IFEval and long text: 41 samples, none above 25 % of a core; the busiest was 20.0 % (a Python one-liner that opens a local web address; I did not trace which program starts it), all others were at most 1.7 % (kernel threads). GPQA 30 with thinking: 57 samples, all of them kernel threads, at most 1.7 % of a core. Nothing was repeated. The card was empty (2 MiB in use) before the server started. Not published: the quiet-check tables; the summaries, logs, contracts and answer files of the tests are published.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-nvfp4-dflash2-tensorfold-1.2.7-ash-recipe-5090-1111.json --parts mmlupro --job ash27nvfp4-5090-127-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-nvfp4-dflash2-tensorfold-1.2.7-ash-recipe-5090-1111.json --parts gpqa --job ash27nvfp4-5090-127-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-nvfp4-dflash2-tensorfold-1.2.7-ash-recipe-5090-1111.json --parts ifeval --job ash27nvfp4-5090-127-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-nvfp4-dflash2-tensorfold-1.2.7-ash-recipe-5090-1111.json --parts ruler --job ash27nvfp4-5090-127-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-nvfp4-dflash2-tensorfold-1.2.7-ash-recipe-5090-1111.json --parts think --job ash27nvfp4-5090-127-1111

How much context fits

Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed at the largest size was not measured. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. Measured 2026-11-11. Raw files: result of every step · log · contract.

Note from the run: The largest size tested is 131,072 tokens (128K), the top step of the series. It loaded and answered, so it is not necessarily the most this card could hold. The model's own window is 262,144 tokens; I did not test sizes above 131,072 tokens. How this differs from the speed and quality tests: the memory test starts the server once per size with the same command plus --context N (the start script takes N from an environment variable); the server's default window of the speed and quality tests of this page is 32,768 tokens. At 131,072 tokens it is the command of the page of the same setup with the larger window, whose start line reads 1 stream at once. The six server logs of this test are not published: they hold only lines of our start script, not the server's start text, so the number of streams at the other windows is not shown. Peak memory is the memory in use on the whole card (nvidia-smi; it was empty before each start, 2 MiB). The tile 'Peak memory' shows the value of the largest size, 29.99 GiB at 128K. The values are 26.95 GiB at 4K, 27.71 GiB at 8K, 27.73 GiB at 16K, 29.12 GiB at 32K, 25.99 GiB at 64K, 29.99 GiB at 128K. They do not rise in line with the window, and I did not log the streams at the smaller windows, so I cannot say why. Decode speed at these sizes is not measured (the server returns no timings).

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-nvfp4-dflash2-tensorfold-1.2.7-ash-recipe-5090-1111.json --parts fit --job ash27nvfp4-5090-127-1111

Also made this possible

• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• Quant (NVFP4, NVIDIA Model Optimizer, revision 482ca0f3)NVIDIAHugging Face GitHub Website
• Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)Inco AI@inco_ai Hugging Face GitHub Website
• Speculative decoding method (DFlash) and the mirror we loadedz-lab@zhijianliu_ GitHub Hugging Face

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one GPU, the whole model in video memory). Host: AMD Ryzen 9 9950X, 96 GB RAM installed (92 GiB usable). Driver 610.43.02, kernel 6.12.95+deb13-amd64

Model

nvidia/Qwen3.8-27B-NVFP4 @482ca0f3 (ModelOpt NVFP4 MLP and head, FP8 attention and DeltaNet projections, bf16 embedding and gates) + z-lab/Qwen3.8-27B-DFlash2 @50307d4c (drafter)

Quant (NVFP4, NVIDIA Model Optimizer, revision 482ca0f3)

huggingface.co/nvidia/Qwen3.8-27B-NVFP4

Draft model (DFlash2, revision 50307d4c)

huggingface.co/z-lab/Qwen3.8-27B-DFlash2

Engine

TensorFold 1.2.7, official release for x86_64 Linux (native CUDA, sm_120), NVIDIA's NVFP4 checkpoint with DFlash2 drafts, window 32,768 tokens, 3 streams chosen by the server — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.7

Context window

32,768 tokens: how much text the model can see at once

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8132, only from this computer.

   Copy

   tensorfold-native serve nvidia/Qwen3.8-27B-NVFP4 \
     --drafter z-lab/Qwen3.8-27B-DFlash2 \
     --name qwen38-27b-nvfp4 \
     --host 127.0.0.1 \
     --port 8132

   What the parts mean

   • --host 127.0.0.1who can connect: only this computer
   • --port 8132the network port the server listens on
   • --drafter z-lab/Qwen3.8-27B-DFlash2the draft model: it guesses the next tokens and the main model checks them; without it the model decodes plain
   • --name qwen38-27b-nvfp4the model ID the server advertises to clients

TensorFold native (Zig) 1.2.7 official release archive tensorfold-1.2.7-linux-x86_64.tar.gz (sha256 319f548b), CUDA (chip nvidia-sm120), DFlash2 drafts, server default window 32768, 1x RTX 5090 (sm_120); tensorfold-native serve nvidia/Qwen3.8-27B-NVFP4 --drafter z-lab/Qwen3.8-27B-DFlash2 --name qwen38-27b-nvfp4 --host 127.0.0.1 --port 8132 (recipe: tensorfold pull nvidia/Qwen3.8-27B-NVFP4 z-lab/Qwen3.8-27B-DFlash2; tensorfold serve nvidia/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8080, docs/recipes/qwen3.8-27b.md:248-251; I started it with --drafter z-lab/Qwen3.8-27B-DFlash2 (README.md:323), a name, a local host and port)