BobClawblaw's recipe on one RTX 5090: Qwen3.8-27B (EXL3 4.00 bpw, DFlash2 drafts) on TensorFold 1.2.7, the recipe's window of 262,144 tokens

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds; long prompts (later supplementary run of 2026-11-12, same setup): prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer · speed result · speed contract · log.

Requests at once: the server has 4 slots (the server's own choice, startup line of the probe Nov 12, 2026). C8 exceeds the slots: the extra requests wait in the queue, and they were measured anyway. What ran. BobClawblaw's recipe for the RTX 5090 (qwen38-27b-vision-rtx-5090 at commit 352e9b6f, MIT): turboderp's EXL3 pack of Qwen3.8-27B at 4.00 bpw (turboderp/Qwen3.8-27B-exl3, branch 4.00bpw, revision 113cf7ab; 6-bit head) with z-lab's DFlash2 drafts (revision 50307d4c, pinned in recipe.toml lines 6 to 10), an fp8 KV cache and the vision tower loaded, started with the recipe's own launch line (serve.sh lines 26 to 41). Both files were checked one by one against Hugging Face before the start. Where this run differs from the recipe. (1) Engine: we used the official TensorFold release 1.2.7 (tensorfold-1.2.7-linux-x86_64.tar.gz, published 11 Nov 2026, sha256 319f548b; binary tensorfold-native sha256 8e5729ba; no patch, nothing built) instead of the maker's own build (TensorFold 1.2.4 plus the two commits of pull request #2626, recipe.toml lines 21 to 33). The flag --instruct-sampling of that pull request is part of the official release since 1.2.6 (CHANGELOG of 1.2.7, line 117). (2) The launch line uses --host 127.0.0.1 instead of 0.0.0.0 (our local measuring setup). (3) Image input is not tested. The JPEG and WebP libraries (libturbojpeg 3.1.4.1, libwebpdecoder 1.6.0), which the recipe's setup.sh builds (lines 46 to 60), are not in the release archive; the server prints two warnings about them at start. They affect only JPEG and WebP image input, not the text tests on this page. Sampling. The recipe serves the model card's no-thinking set with --instruct-sampling card (temperature 0.7, top_p 0.8, top_k 20, presence penalty 1.5; README lines 108 to 128), and the speed test sends the same values. The recipe's README and TensorFold issue #2645 note that this presence penalty lowers the DFlash2 draft acceptance (in the maker's measurement by about 5 % on average). Against the maker's own numbers. The README lists decode numbers from the maker's own test (his harness, greedy decoding, his build of TensorFold 1.2.4 plus #2626, 8 to 9 Nov 2026). That is a different method and a different engine build from ours; those numbers are not measured by us and are not shown next to ours. Window and streams. The window is the recipe's 262,144 tokens and the launch line uses --parallel auto. With this window the server chose 4 streams itself: its start line reads '4 streams at once, the engine's own at a 262144-token window (9.09 GiB, in the model's), 3 more taking 322 pages of 256 rows (2.55 GiB) as they need them' (the model takes 18.16 GiB). All context steps fit this window; the step called 128K was measured at 131,072 tokens. Requests at once. The server has 4 streams. '2', '4' and '8 at once' send that many requests, and any above the stream count wait for a free stream; the tool logs them as measured anyway, so those rows include the waiting (the 8 requests of '8 at once' are two waves on 4 streams). The long prompts at 16K to 128K come from a supplementary run of the same setup with a fresh server start (raw folder of the supplement on this page).

python3 speed2.py --base-url http://127.0.0.1:45325/v1 --label bob400exl3-5090-127-1211-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/bob400exl3-5090-127-1211-speed-r0/bob400exl3-5090-127-1211-speed-r0/speed2 --contract $RUN/bob400exl3-5090-127-1211-speed-r0/bob400exl3-5090-127-1211-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/bob400-1211/models/Qwen3.8-27B-exl3-4.00bpw/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source Bob's recipe README.md:108-128 (Sampling defaults, revision 352e9b6f): thinking replies temperature 1.0, top_p 0.95, top_k 20 (the pack's generation_config.json and the server default); instruct (thinking off) temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5, served by --instruct-sampling card (serve.sh:40); the same values as the Qwen card (https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 and 253, local copy not published line 252-253); min_p 0.0 and thinking presence_penalty 0.0 are the Qwen card's values for the parameters the recipe leaves open --model qwen38-27b --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 4 --server-slots-source Bob's recipe README.md:468-471 (EXL3 pack: three streams at 262,144 on 1.2.4 + #2626) and serve.sh:37 (--parallel auto); TensorFold v1.2.7 README.md:345 (--parallel auto asks for up to eight streams as memory and the engine allow); streams = the server's own choice, read from the startup line at the probe start Nov 12, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 18.16 GiB; 4 streams at once, the engine's own at a 262144-token window (9.09 GiB, in the model's), 3 more taking 322 pages of 256 rows (2.55 GiB) as they need them, 23 MiB fixed each, of 11.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; ' --server-ctx 262144 --server-ctx-source Bob's launch: the recipe window. Startup line of the probe Nov 12, 2026 (release 1.2.7, RTX 5090, Bob's launch): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 18.16 GiB; 4 streams at once, the engine's own at a 262144-token window (9.09 GiB, in the model's), 3 more taking 322 pages of 256 rows (2.55 GiB) as they need them, 23 MiB fixed each, of 11.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows' and '[tensorfold] serving qwen38-27b at http://127.0.0.1:30000/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; thinking off: temperature 0.7, top_k 20, top_p 0.8, presence_penalty 1.5; drafts: DFlash2 ($HOME/bob400-1211/models/Qwen3.8-27B-DFlash2) + copies; context: 262144; loaded in 29.2s)'. Window source: recipe serve.sh:37 and README.md:591 (CONTEXT 262144).

# Long prompts: supplementary run of 2026-11-12 (raw folder bob400exl3-5090-127-1211-ctx-speed-r0-ctx)
python3 speed2.py --base-url http://127.0.0.1:44163/v1 --label bob400exl3-5090-127-1211-ctx-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/bob400exl3-5090-127-1211-ctx-speed-r0/bob400exl3-5090-127-1211-ctx-speed-r0/speed2 --contract $RUN/bob400exl3-5090-127-1211-ctx-speed-r0/bob400exl3-5090-127-1211-ctx-speed-r0/speed2/speed-contract.json --parts ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/bob400-1211/models/Qwen3.8-27B-exl3-4.00bpw/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source Bob's recipe README.md:108-128 (Sampling defaults, revision 352e9b6f): thinking replies temperature 1.0, top_p 0.95, top_k 20 (the pack's generation_config.json and the server default); instruct (thinking off) temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5, served by --instruct-sampling card (serve.sh:40); the same values as the Qwen card (https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 and 253, local copy not published line 252-253); min_p 0.0 and thinking presence_penalty 0.0 are the Qwen card's values for the parameters the recipe leaves open --model qwen38-27b --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 4 --server-slots-source Bob's recipe README.md:468-471 (EXL3 pack: three streams at 262,144 on 1.2.4 + #2626) and serve.sh:37 (--parallel auto); TensorFold v1.2.7 README.md:345 (--parallel auto asks for up to eight streams as memory and the engine allow); streams = the server's own choice, read from the startup line at the probe start Nov 12, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 18.16 GiB; 4 streams at once, the engine's own at a 262144-token window (9.09 GiB, in the model's), 3 more taking 322 pages of 256 rows (2.55 GiB) as they need them, 23 MiB fixed each, of 11.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; ' --server-ctx 262144 --server-ctx-source Bob's launch: the recipe window. Startup line of the probe Nov 12, 2026 (release 1.2.7, RTX 5090, Bob's launch): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 18.16 GiB; 4 streams at once, the engine's own at a 262144-token window (9.09 GiB, in the model's), 3 more taking 322 pages of 256 rows (2.55 GiB) as they need them, 23 MiB fixed each, of 11.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows' and '[tensorfold] serving qwen38-27b at http://127.0.0.1:30000/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; thinking off: temperature 0.7, top_k 20, top_p 0.8, presence_penalty 1.5; drafts: DFlash2 ($HOME/bob400-1211/models/Qwen3.8-27B-DFlash2) + copies; context: 262144; loaded in 29.2s)'. Window source: recipe serve.sh:37 and README.md:591 (CONTEXT 262144). --supplement-of bob400exl3-5090-127-1211

How well it answers

Thinking off per request except the GPQA 30 test; the Qwen model card's sampling, seed 42, fixed lists. Raw numbers only, no overall score.

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-12.

Note from the run: MMLU-Pro, GPQA Diamond, IFEval and the long-text tests ran with thinking off per request, the Qwen model card's no-thinking sampling and seed 42, fixed lists. GPQA with thinking (30 questions) ran with thinking on and the thinking sampling; 7 of its 30 answers were cut off at the limit of 16,384 tokens and count as wrong, and the median token count includes those cut-off answers at 16,384 tokens each. LiveCodeBench was not run on this card (it is on the page with the window of 131,072 tokens). The numbers are raw numbers with their 95 % margin, not an overall score.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-4.00bpw-bob-recipe-tensorfold-1.2.7-5090-1211.json --parts mmlupro --job bob400exl3-5090-127-1211

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-4.00bpw-bob-recipe-tensorfold-1.2.7-5090-1211.json --parts gpqa --job bob400exl3-5090-127-1211

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-4.00bpw-bob-recipe-tensorfold-1.2.7-5090-1211.json --parts ifeval --job bob400exl3-5090-127-1211

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-4.00bpw-bob-recipe-tensorfold-1.2.7-5090-1211.json --parts ruler --job bob400exl3-5090-127-1211

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-4.00bpw-bob-recipe-tensorfold-1.2.7-5090-1211.json --parts ruler --job bob400exl3-5090-127-1211

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-4.00bpw-bob-recipe-tensorfold-1.2.7-5090-1211.json --parts think --job bob400exl3-5090-127-1211

How much context fits

Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed at the largest size was not measured. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. Measured 2026-11-12. Raw files: result of every step · log · contract.

Note from the run: The list of sizes ends at 128K (131,072 tokens), so this is the largest size tested, not the largest size the setup can load: the recipe's own window of 262,144 tokens loaded and ran in all other parts of this page.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-4.00bpw-bob-recipe-tensorfold-1.2.7-5090-1211.json --parts fit --job bob400exl3-5090-127-1211

Also made this possible

• Engine (TensorFold official release 1.2.7; the recipe itself names the maker's own build)Ash Hart@ashxhart GitHub Hugging Face Website
• Quant (EXL3 4.00 bpw, 6-bit head, revision 113cf7ab)turboderp@turboderp_ Hugging Face GitHub
• ModelQwen@Alibaba_Qwen Hugging Face GitHub Website
• Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)Inco AI@inco_ai Hugging Face GitHub Website
• Draft model repository we loaded (z-lab/Qwen3.8-27B-DFlash2)z-lab (Zhijian Liu)@zhijianliu_ Hugging Face GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one machine), driver 610.43.02, kernel 6.12.95 (Debian 13)

Model

turboderp/Qwen3.8-27B-exl3 @113cf7ab (EXL3 4.00 bpw, 6-bit head, mul1; vision tower loaded) + z-lab/Qwen3.8-27B-DFlash2 @50307d4c (drafter); recipe by BobClawblaw (qwen38-27b-vision-rtx-5090 @352e9b6f) on the official TensorFold 1.2.7

Quant (EXL3 4.00 bpw, 6-bit head, revision 113cf7ab)

huggingface.co/turboderp/Qwen3.8-27B-exl3/tree/113cf7ab958054860e43fb7f3063b1af19171095

Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)

huggingface.co/incoai/Qwen3.8-27B-DFlash2

Draft model repository we loaded (z-lab/Qwen3.8-27B-DFlash2)

huggingface.co/z-lab/Qwen3.8-27B-DFlash2/tree/50307d4c4cde6860d4eee73e2547cd786fe8e8a4

Engine

Recipe qwen38-27b-vision-rtx-5090 at commit 352e9b6f, run on the official TensorFold release 1.2.7 instead of the maker's own build — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.7

Recipe (qwen38-27b-vision-rtx-5090, commit 352e9b6f, MIT)

github.com/BobClawblaw/qwen38-27b-vision-rtx-5090/tree/352e9b6f904dc025629dd48868063fcd08600f14 by BobClawblaw

Steps

1. Get the recipe, the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 30000, also from other machines in your network.

   Copy

   Recipe (serve.sh lines 26 to 41): \
     TENSORFOLD_CUDA_KERNELS=native \
     TF_QWEN_KV_PAGES=256 tensorfold-native serve <models>/Qwen3.8-27B-exl3-4.00bpw \
     --drafter <models>/Qwen3.8-27B-DFlash2 \
     --name qwen38-27b \
     --alias qwen38-nvfp4 \
     --host 0.0.0.0 \
     --port 30000 \
     --context 262144 \
     --parallel auto \
     --kv-cache fp8 \
     --max-tokens 65536 \
     --thinking-budget 49152 \
     --instruct-sampling card \
     --vision. Our start (official release 1.2.7, binary tensorfold-native of the archive): \
     TENSORFOLD_CUDA_KERNELS=native \
     TF_QWEN_KV_PAGES=256 tensorfold-native serve <models>/Qwen3.8-27B-exl3-4.00bpw \
     --drafter <models>/Qwen3.8-27B-DFlash2 \
     --name qwen38-27b \
     --alias qwen38-nvfp4 \
     --host 127.0.0.1 \
     --port 30000 \
     --context 262144 \
     --parallel auto \
     --kv-cache fp8 \
     --max-tokens 65536 \
     --thinking-budget 49152 \
     --instruct-sampling card \
     --vision (host changed to 127.0.0.1, nothing else).

   What the parts mean

   • --host 0.0.0.0who can connect: also other machines in your network. Careful: any machine in your network can then reach the server; use 127.0.0.1 if only this computer should
   • --port 30000the network port the server listens on
   • --drafter <models>/Qwen3.8-27B-DFlash2the draft model: it guesses the next tokens and the main model checks them; without it the model decodes plain
   • --name qwen38-27bthe model ID the server advertises to clients
   • --alias qwen38-nvfp4the model name clients use in their requests

TensorFold native (Zig) 1.2.7 official release archive tensorfold-1.2.7-linux-x86_64.tar.gz (sha256 319f548b), CUDA sm_120, DFlash2 drafts, --kv-cache fp8, --vision, recipe window 262144, 1x RTX 5090 (sm_120); Recipe (serve.sh lines 26 to 41): TENSORFOLD_CUDA_KERNELS=native TF_QWEN_KV_PAGES=256 tensorfold-native serve <models>/Qwen3.8-27B-exl3-4.00bpw --drafter <models>/Qwen3.8-27B-DFlash2 --name qwen38-27b --alias qwen38-nvfp4 --host 0.0.0.0 --port 30000 --context 262144 --parallel auto --kv-cache fp8 --max-tokens 65536 --thinking-budget 49152 --instruct-sampling card --vision. Our start (official release 1.2.7, binary tensorfold-native of the archive): TENSORFOLD_CUDA_KERNELS=native TF_QWEN_KV_PAGES=256 tensorfold-native serve <models>/Qwen3.8-27B-exl3-4.00bpw --drafter <models>/Qwen3.8-27B-DFlash2 --name qwen38-27b --alias qwen38-nvfp4 --host 127.0.0.1 --port 30000 --context 262144 --parallel auto --kv-cache fp8 --max-tokens 65536 --thinking-budget 49152 --instruct-sampling card --vision (host changed to 127.0.0.1, nothing else).