BobClawblaw's recipe on one RTX 5090: Qwen3.8-27B (EXL3 4.00 bpw, DFlash2 drafts) on TensorFold 1.2.7, window 131,072 tokens and one stream

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds; long prompts (later supplementary run of 2026-11-12, same setup): prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer · speed result · speed contract · log.

Requests at once: the server has 1 slot (--parallel 1, named deviation of this job; startup line of the probe Nov 12, 2026). C2, C4, C8 exceed the slots: the extra requests wait in the queue, and they were measured anyway. What ran. BobClawblaw's recipe for the RTX 5090 (qwen38-27b-vision-rtx-5090 at commit 352e9b6f, MIT): turboderp's EXL3 pack of Qwen3.8-27B at 4.00 bpw (turboderp/Qwen3.8-27B-exl3, branch 4.00bpw, revision 113cf7ab; 6-bit head) with z-lab's DFlash2 drafts (revision 50307d4c, pinned in recipe.toml lines 6 to 10), an fp8 KV cache and the vision tower loaded, started with the recipe's own launch line (serve.sh lines 26 to 41). Both files were checked one by one against Hugging Face before the start. Where this run differs from the recipe. (1) Engine: we used the official TensorFold release 1.2.7 (tensorfold-1.2.7-linux-x86_64.tar.gz, published 11 Nov 2026, sha256 319f548b; binary tensorfold-native sha256 8e5729ba; no patch, nothing built) instead of the maker's own build (TensorFold 1.2.4 plus the two commits of pull request #2626, recipe.toml lines 21 to 33). The flag --instruct-sampling of that pull request is part of the official release since 1.2.6 (CHANGELOG of 1.2.7, line 117). (2) The launch line uses --host 127.0.0.1 instead of 0.0.0.0 (our local measuring setup). (3) Image input is not tested. The JPEG and WebP libraries (libturbojpeg 3.1.4.1, libwebpdecoder 1.6.0), which the recipe's setup.sh builds (lines 46 to 60), are not in the release archive; the server prints two warnings about them at start. They affect only JPEG and WebP image input, not the text tests on this page. Sampling. The recipe serves the model card's no-thinking set with --instruct-sampling card (temperature 0.7, top_p 0.8, top_k 20, presence penalty 1.5; README lines 108 to 128), and the speed test sends the same values. The recipe's README and TensorFold issue #2645 note that this presence penalty lowers the DFlash2 draft acceptance (in the maker's measurement by about 5 % on average). Against the maker's own numbers. The README lists decode numbers from the maker's own test (his harness, greedy decoding, his build of TensorFold 1.2.4 plus #2626, 8 to 9 Nov 2026). That is a different method and a different engine build from ours; those numbers are not measured by us and are not shown next to ours. Window and streams. This run has two named deviations from the launch line, both chosen by us, not by the maker: --context 131072 instead of 262,144 and --parallel 1 instead of auto. Reason: this card measures the window of 131,072 tokens with one stream, the setting we also use for the other 32 GB pages with a 131,072-token window, so they can be read side by side; the recipe's README names CONTEXT and PARALLEL as knobs (lines 587 to 593). In a probe start with --parallel auto at this window the server chose 8 streams; with --parallel 1 its start line reads '1 stream at once, the engine's own at a 131072-token window (5.02 GiB, in the model's), 0 more taking 0 pages of 256 rows (0.00 GiB)'. The 128K step could not use 131,072 prompt tokens (with 512 reply tokens it would exceed the window), so the test takes the window minus 1,024, which is 130,048 tokens (its rule, valid from 126,976); the step called 128K was measured at 130,048 tokens. Requests at once. The server has 1 stream. '2', '4' and '8 at once' send that many requests, and any above the stream count wait for a free stream; the tool logs them as measured anyway, so those rows include the waiting. Spread. The spread of the code runs (CV) is above 15 % on this card (16.7 %); the speed test asks for a new measurement above that value, and this card has no repeat run, so the value is shown as measured. The long prompts at 16K to 128K come from a supplementary run of the same setup with a fresh server start (raw folder of the supplement on this page).

python3 speed2.py --base-url http://127.0.0.1:40629/v1 --label bob400exl3-5090-127-ctx131072-1211-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/bob400exl3-5090-127-ctx131072-1211-speed-r0/bob400exl3-5090-127-ctx131072-1211-speed-r0/speed2 --contract $RUN/bob400exl3-5090-127-ctx131072-1211-speed-r0/bob400exl3-5090-127-ctx131072-1211-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/bob400-1211/models/Qwen3.8-27B-exl3-4.00bpw/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source Bob's recipe README.md:108-128 (Sampling defaults, revision 352e9b6f): thinking replies temperature 1.0, top_p 0.95, top_k 20 (the pack's generation_config.json and the server default); instruct (thinking off) temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5, served by --instruct-sampling card (serve.sh:40); the same values as the Qwen card (https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 and 253, local copy not published line 252-253); min_p 0.0 and thinking presence_penalty 0.0 are the Qwen card's values for the parameters the recipe leaves open --model qwen38-27b --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source Bob's recipe README.md:468-471 (EXL3 pack: three streams at 262,144 on 1.2.4 + #2626) and serve.sh:37 (--parallel auto); TensorFold v1.2.7 README.md:345 (--parallel auto asks for up to eight streams as memory and the engine allow); streams = the server's own choice, read from the startup line at the probe start Nov 12, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 18.16 GiB; 1 stream at once, the engine's own at a 131072-token window (5.02 GiB, in the model's), 0 more taking 0 pages of 256 rows (0.00 GiB) as they need them, 23 MiB fixed each, of 11.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own' --server-ctx 131072 --server-ctx-source DEVIATION named: --context 131072 (see _comment). Startup line of the probe Nov 12, 2026 (release 1.2.7, RTX 5090, Bob's launch): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 18.16 GiB; 1 stream at once, the engine's own at a 131072-token window (5.02 GiB, in the model's), 0 more taking 0 pages of 256 rows (0.00 GiB) as they need them, 23 MiB fixed each, of 11.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a' and '[tensorfold] serving qwen38-27b at http://127.0.0.1:30000/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; thinking off: temperature 0.7, top_k 20, top_p 0.8, presence_penalty 1.5; drafts: DFlash2 ($HOME/bob400-1211/models/Qwen3.8-27B-DFlash2) + copies; context: 131072; loaded in 21.7s)'. Window source: recipe serve.sh:37 and README.md:591 (CONTEXT 262144).

# Long prompts: supplementary run of 2026-11-12 (raw folder bob400exl3-5090-127-ctx131072-1211-ctx-speed-r0-ctx)
python3 speed2.py --base-url http://127.0.0.1:44313/v1 --label bob400exl3-5090-127-ctx131072-1211-ctx-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/bob400exl3-5090-127-ctx131072-1211-ctx-speed-r0/bob400exl3-5090-127-ctx131072-1211-ctx-speed-r0/speed2 --contract $RUN/bob400exl3-5090-127-ctx131072-1211-ctx-speed-r0/bob400exl3-5090-127-ctx131072-1211-ctx-speed-r0/speed2/speed-contract.json --parts ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/bob400-1211/models/Qwen3.8-27B-exl3-4.00bpw/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source Bob's recipe README.md:108-128 (Sampling defaults, revision 352e9b6f): thinking replies temperature 1.0, top_p 0.95, top_k 20 (the pack's generation_config.json and the server default); instruct (thinking off) temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5, served by --instruct-sampling card (serve.sh:40); the same values as the Qwen card (https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 and 253, local copy not published line 252-253); min_p 0.0 and thinking presence_penalty 0.0 are the Qwen card's values for the parameters the recipe leaves open --model qwen38-27b --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source Bob's recipe README.md:468-471 (EXL3 pack: three streams at 262,144 on 1.2.4 + #2626) and serve.sh:37 (--parallel auto); TensorFold v1.2.7 README.md:345 (--parallel auto asks for up to eight streams as memory and the engine allow); streams = the server's own choice, read from the startup line at the probe start Nov 12, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 18.16 GiB; 1 stream at once, the engine's own at a 131072-token window (5.02 GiB, in the model's), 0 more taking 0 pages of 256 rows (0.00 GiB) as they need them, 23 MiB fixed each, of 11.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own' --server-ctx 131072 --server-ctx-source DEVIATION named: --context 131072 (see _comment). Startup line of the probe Nov 12, 2026 (release 1.2.7, RTX 5090, Bob's launch): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 18.16 GiB; 1 stream at once, the engine's own at a 131072-token window (5.02 GiB, in the model's), 0 more taking 0 pages of 256 rows (0.00 GiB) as they need them, 23 MiB fixed each, of 11.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a' and '[tensorfold] serving qwen38-27b at http://127.0.0.1:30000/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; thinking off: temperature 0.7, top_k 20, top_p 0.8, presence_penalty 1.5; drafts: DFlash2 ($HOME/bob400-1211/models/Qwen3.8-27B-DFlash2) + copies; context: 131072; loaded in 21.7s)'. Window source: recipe serve.sh:37 and README.md:591 (CONTEXT 262144). --supplement-of bob400exl3-5090-127-ctx131072-1211

How well it answers

Thinking off per request, the Qwen model card's no-thinking sampling, seed 42, fixed lists. Raw numbers only, no overall score.

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-12.

Note from the run: LiveCodeBench, the long-text tests ran with thinking off per request, the Qwen model card's no-thinking sampling and seed 42, fixed lists, at the window of 131,072 tokens with one stream. 1 of the 40 LiveCodeBench answers was cut off at its limit and counts as wrong. MMLU-Pro, GPQA Diamond, IFEval and GPQA with thinking were not run on this card (they are on the page with the recipe's own window). The numbers are raw numbers with their 95 % margin, not an overall score.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-4.00bpw-bob-recipe-tensorfold-1.2.7-5090-ctx131072-1211.json --parts lcb --job bob400exl3-5090-127-ctx131072-1211

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-4.00bpw-bob-recipe-tensorfold-1.2.7-5090-ctx131072-1211.json --parts ruler --job bob400exl3-5090-127-ctx131072-1211

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-4.00bpw-bob-recipe-tensorfold-1.2.7-5090-ctx131072-1211.json --parts ruler --job bob400exl3-5090-127-ctx131072-1211

Also made this possible

• Engine (TensorFold official release 1.2.7; the recipe itself names the maker's own build)Ash Hart@ashxhart GitHub Hugging Face Website
• Quant (EXL3 4.00 bpw, 6-bit head, revision 113cf7ab)turboderp@turboderp_ Hugging Face GitHub
• ModelQwen@Alibaba_Qwen Hugging Face GitHub Website
• Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)Inco AI@inco_ai Hugging Face GitHub Website
• Draft model repository we loaded (z-lab/Qwen3.8-27B-DFlash2)z-lab (Zhijian Liu)@zhijianliu_ Hugging Face GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one machine), driver 610.43.02, kernel 6.12.95 (Debian 13)

Model

turboderp/Qwen3.8-27B-exl3 @113cf7ab (EXL3 4.00 bpw, 6-bit head, mul1; vision tower loaded) + z-lab/Qwen3.8-27B-DFlash2 @50307d4c (drafter); recipe by BobClawblaw (qwen38-27b-vision-rtx-5090 @352e9b6f) on the official TensorFold 1.2.7

Quant (EXL3 4.00 bpw, 6-bit head, revision 113cf7ab)

huggingface.co/turboderp/Qwen3.8-27B-exl3/tree/113cf7ab958054860e43fb7f3063b1af19171095

Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)

huggingface.co/incoai/Qwen3.8-27B-DFlash2

Draft model repository we loaded (z-lab/Qwen3.8-27B-DFlash2)

huggingface.co/z-lab/Qwen3.8-27B-DFlash2/tree/50307d4c4cde6860d4eee73e2547cd786fe8e8a4

Engine

Recipe qwen38-27b-vision-rtx-5090 at commit 352e9b6f, run on the official TensorFold release 1.2.7, with --context 131072 and --parallel 1 — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.7

Recipe (qwen38-27b-vision-rtx-5090, commit 352e9b6f, MIT)

github.com/BobClawblaw/qwen38-27b-vision-rtx-5090/tree/352e9b6f904dc025629dd48868063fcd08600f14 by BobClawblaw

Steps

1. Get the recipe, the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 30000, also from other machines in your network.

   Copy

   Recipe (serve.sh lines 26 to 41): \
     TENSORFOLD_CUDA_KERNELS=native \
     TF_QWEN_KV_PAGES=256 tensorfold-native serve <models>/Qwen3.8-27B-exl3-4.00bpw \
     --drafter <models>/Qwen3.8-27B-DFlash2 \
     --name qwen38-27b \
     --alias qwen38-nvfp4 \
     --host 0.0.0.0 \
     --port 30000 \
     --context 262144 \
     --parallel auto \
     --kv-cache fp8 \
     --max-tokens 65536 \
     --thinking-budget 49152 \
     --instruct-sampling card \
     --vision. Our start (official release 1.2.7, binary tensorfold-native of the archive): \
     TENSORFOLD_CUDA_KERNELS=native \
     TF_QWEN_KV_PAGES=256 tensorfold-native serve <models>/Qwen3.8-27B-exl3-4.00bpw \
     --drafter <models>/Qwen3.8-27B-DFlash2 \
     --name qwen38-27b \
     --alias qwen38-nvfp4 \
     --host 127.0.0.1 \
     --port 30000 \
     --context 131072 \
     --parallel 1 \
     --kv-cache fp8 \
     --max-tokens 65536 \
     --thinking-budget 49152 \
     --instruct-sampling card \
     --vision (host changed to 127.0.0.1, window and stream count changed, see the page).

   What the parts mean

   • --parallel 1how many requests it answers at the same time
   • --host 0.0.0.0who can connect: also other machines in your network. Careful: any machine in your network can then reach the server; use 127.0.0.1 if only this computer should
   • --port 30000the network port the server listens on
   • --drafter <models>/Qwen3.8-27B-DFlash2the draft model: it guesses the next tokens and the main model checks them; without it the model decodes plain
   • --name qwen38-27bthe model ID the server advertises to clients
   • --alias qwen38-nvfp4the model name clients use in their requests

TensorFold native (Zig) 1.2.7 official release archive tensorfold-1.2.7-linux-x86_64.tar.gz (sha256 319f548b), CUDA sm_120, DFlash2 drafts, --kv-cache fp8, --vision, window 131072 (named deviation --context 131072, --parallel 1), 1x RTX 5090 (sm_120); Recipe (serve.sh lines 26 to 41): TENSORFOLD_CUDA_KERNELS=native TF_QWEN_KV_PAGES=256 tensorfold-native serve <models>/Qwen3.8-27B-exl3-4.00bpw --drafter <models>/Qwen3.8-27B-DFlash2 --name qwen38-27b --alias qwen38-nvfp4 --host 0.0.0.0 --port 30000 --context 262144 --parallel auto --kv-cache fp8 --max-tokens 65536 --thinking-budget 49152 --instruct-sampling card --vision. Our start (official release 1.2.7, binary tensorfold-native of the archive): TENSORFOLD_CUDA_KERNELS=native TF_QWEN_KV_PAGES=256 tensorfold-native serve <models>/Qwen3.8-27B-exl3-4.00bpw --drafter <models>/Qwen3.8-27B-DFlash2 --name qwen38-27b --alias qwen38-nvfp4 --host 127.0.0.1 --port 30000 --context 131072 --parallel 1 --kv-cache fp8 --max-tokens 65536 --thinking-budget 49152 --instruct-sampling card --vision (host changed to 127.0.0.1, window and stream count changed, see the page).