Bob's Qwen3.8-27B recipe on Ash's official TensorFold 1.2.6 and one RTX 5090 (MLX 4-bit, DFlash2 drafts)

How fast it runs

Long prompts

Requests at once, deviation from the recipe (window of 32,768 tokens)

This block is a separate run with another setup. It is not a second reading of the table above, and its numbers do not belong to the rows above it. The recipe's window is 262,144 tokens and the server then has 1 stream, so in the table above 2, 4 and 8 requests at once only queue. For this run I set the recipe's own CONTEXT variable to 32,768 tokens, which is the window that the release README names as the default for Qwen3.8-27B (README line 307), and changed nothing else: same binary, same serve.sh, same two checkpoints, same test settings. At that window the server chose its streams itself. Its start line, as the speed contract of this run quotes it (field server_slots_source; the server log itself is not published), reads: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 17.67 GiB; 6 streams at once, 1.98 GiB each at a 32768-token window (fp8 caches), of 12.2 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' (raw folder tf5090-rel126-1111-conc-speed-r0-conc).

With 6 streams the levels 1, 2 and 4 are inside the streams, and '8 at once' sends 8 requests to 6 streams, so 2 of them wait. The tool logs that level as measured anyway, so the row for 8 at once measures the waiting as well. The total is 210.6 tok/s with 1 request, 314.1 with 2, 400.1 with 4 and 406.5 with 8, while the speed of one request falls from 210.6 to 68.3 tok/s. Two rounds per level, 512 tokens per request, the prose prompt, the median of the rounds, seed 42, the same test settings as the table above.

The rows of the window of 262,144 tokens (recipe as written, 1 stream) stay in the table above. I show this series because the recipe's own setting cannot run more than one request at a time, and I give the window and the stream count next to it so that the two series are never read as one.

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds; long prompts (later supplementary run of 2026-11-11, same setup): prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests. Measured 2026-11-11. Raw files: speed result · speed contract · log · long answer · speed result · speed contract · log · speed result · speed contract · log.

Requests at once: the server has 1 slot (from the author's recipe). C2, C4, C8 exceed the slots: the extra requests wait in the queue, and they were measured anyway. The speed test ran the settings of BobClawblaw's recipe (repository qwen38-27b-vision-rtx-5090, revision 2147ce9d, serve.sh lines 23 to 30) with the binary of the official TensorFold release 1.2.6, the archive tensorfold-1.2.6-linux-x86_64.tar.gz (published 10 Nov 2026, 12:51 UTC, sha256 a8abf9b8, the digest GitHub lists for that file). The binary tensorfold-native inside it has the md5 7c69623c9a71 that the measuring machine recorded. The recipe's setup builds its own engine from a branch of BobClawblaw's fork of TensorFold (recipe.toml lines 10 to 12); I did not build or run that engine, so this page shows the recipe's settings on Ash's official release, not Bob's build. serve.sh ran as written, with HOST=127.0.0.1 and PORT=8888 set from outside and with MODELS and TF_SRC, the folders that serve.sh reads from the environment (lines 6 to 8, 16 and 17), pointing at the two checkpoints and at the unpacked release archive. The folder layout that serve.sh expects (zig-out/native/bin/tensorfold-native and zig-out/native/lib/libtfgrammar.so, lines 22 and 23) was made from the files of the release archive. The command that serve.sh runs with these settings is under Setup. A Linux RTX 5090 is not one of the machines that the release README names for this model: its table of models gives for Qwen3.8-27B, MLX 4-bit under 'Machine and memory scope' the text 'One Mac (M5 Max/M3 Ultra), or one GB10; a 32 GB Mac needs iogpu.wired_limit_mb raised (the startup refusal names the value).' (README line 182), and the README says that its CUDA release images include sm_80, sm_86, sm_120 and sm_121 (line 170). The release's recipe document for this model says that on CUDA the native engine serves the MLX affine checkpoints (2, 3, 4, 5, 6 and 8 bits in groups of 64) with DFlash2 drafts and --kv-cache fp8 (docs/recipes/qwen3.8-27b.md lines 7 to 9), and that CUDA requires DFlash2 unless --no-drafts is set (line 206). The recipe's README names a single RTX 5090 as its machine (lines 3 to 5). This page is my transfer of the recipe to the official release on that machine. It is not a result that Ash or Bob published, and I draw no conclusion beyond what this run shows. Revisions. The recipe pins the model files at revision 22d8d538 and the drafter at revision 50307d4c (recipe.toml lines 5 and 7). Hugging Face lists both as the newest revision of their repositories on 11 Nov 2026 (last changed 1 Oct 2026 and 19 Aug 2026). The folders on the measuring machine were put there before this run, and for this page I did not hash their files against the revisions. The drafter's model card says that z-lab's repository is a mirror of incoai/Qwen3.8-27B-DFlash2 (lines 24 and 25), and the weight file of the two repositories (model.safetensors, 3,848,817,896 bytes, sha256 67fc76d6) has the same size and hash in both file lists. The recipe's repository has later commits than revision 2147ce9d of 22 Oct 2026 (the newest I read, 352e9b6f, is from 9 Nov 2026). I did not run them: this page measures the revision that the setup card pins. Window and streams. serve.sh defaults to a window of 262,144 tokens and --parallel auto (line 27), and I left both as they are. The command cards below quote the source note of the stream count as it was run, and that note says serve.sh:28 for the --parallel flag; the flag is on line 27, and line 28 holds --kv-cache and --prompt-precision. The recipe's README says that PARALLEL auto means as many streams as the window allows, one at 262,144 tokens (line 277), and the tool took the stream count 1 from that line. The release README says that --parallel auto asks for up to eight streams as memory and the engine allow (lines 104 and 345). The server's start text of this run is not among the published files, so the count of 1 is the recipe's statement, not a line I read from the engine. A second series of requests at once, with another window and 6 streams, is shown in its own block below the table of long prompts. It is a deviation from the recipe and is not part of the numbers in this note and the tables above it. Requests at once (this run). With 1 stream, '2 at once', '4 at once' and '8 at once' send more requests than the server has streams, so the extra requests wait in the queue. The tool logs these levels as measured anyway. The speed of one request in the table is its tokens divided by the time from sending to the end of the reply, so for 2, 4 and 8 at once it includes the waiting. The total is 209.9 tok/s with 1 request, 220.9 with 2, 209.0 with 4 and 206.7 with 8, while the speed of one request falls from 209.9 to 48.4 tok/s. Long prompts. All four steps (16K, 32K, 64K, 128K) have their exact length, because the window of 262,144 tokens holds the prompt and the 512 reply tokens. The long-prompt steps were measured twice. The speed run contained them (raw folder tf5090-rel126-1111-speed-r0), and a supplementary run of the same day repeated only these steps (raw folder tf5090-rel126-1111-ctx-speed-r0-ctx, a new server start with the same command). The table shows the supplementary run. In the speed run the tool flagged the decode of the 32K prose step as noisy (spread (CV) of 15.7 %, limit 15 %) and re-measure was its advice, which is why the supplementary run exists. In the supplementary run no step is flagged and no decode, prefill or first-token value has a spread above 7.7 %. The values of the speed run are: 16K: 224.3 tok/s (code) and 217.0 tok/s (prose), prefill 4276.9 tok/s, first token after 3.8 s; 32K: 218.7 tok/s (code) and 208.4 tok/s (prose), prefill 3899.8 tok/s, first token after 8.4 s; 64K: 195.4 tok/s (code) and 192.2 tok/s (prose), prefill 3310.2 tok/s, first token after 19.8 s; 128K: 163.9 tok/s (code) and 155.7 tok/s (prose), prefill 2566.3 tok/s, first token after 51.1 s. The prefill series of the speed run (2K, 8K and 32K, 3 runs each) and the long-prompt steps are separate tests with their own prompts, so the tile 'Prefill at 32K' (4000.5 tok/s, first token after 8.2 s) and the 32K row of the long-prompt table are two measurements. Sampling is the model card's setting for replies without thinking (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5; model card line 253), sent with the request flag for no thinking and seed 42. The chat template thinks by default, and serve.sh passes no sampling flag, so the server's own default is temperature 1.0, top_k 20, top_p 0.95 (start text in the server log of the run at the window of 32,768 tokens; that log is not published). The recipe's README reports its own speed numbers, measured with TensorFold's bench_openai.py (README lines 96 and 305) on the engine build that it pins. This test uses another method (two prompts, medians of two runs, 512 tokens) and the official release, so I compare nothing against those numbers. The headline 275.5 tok/s is the mean of the code prompt (305.3) and the prose prompt (245.7), medians of two runs each (spread (CV) of 7.1 % and 11.3 %). The code prompt has 1,258 and 1,259 tokens and the prose prompt 1,352 and 1,354 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 21.1 % (1,560 of 7,393 drafted tokens, mean accepted length 4.1 tokens). The row 'Decode, window 1' is the decode speed over reply tokens 1 to 256, taken like the headline as the mean of the two prompts' medians. Its two code-prompt runs reached 283.8 and 383.8 tok/s (spread (CV) of 21.2 %) and its two prose-prompt runs 252.2 and 237.8 tok/s (spread (CV) of 4.2 %). The long-answer test asks for one reply of up to 32,768 tokens; the reply had 5,574 tokens and ended with a stop after 16.4 seconds. The release lists an optional drafter for Qwen3.8-27B on CUDA, DSpine-b16, that is used when --drafter names it (release notes). The recipe and this run use the DFlash2 drafter, so the optional one is not part of this run. Foreign load. The load guard (any other process above 25 % of a core for two samples in a row, sampled twice a second) found no other process above its limit in any of the 38 steps of the speed run, the 16 steps of the supplementary run of the long-prompt steps and the 8 steps of the run at the window of 32,768 tokens, so no step was repeated. The highest other process in any step was the Docker daemon (dockerd) at 16.0 % of a core; one step of the speed run (the 64K code step, first run) shows a Python process at 6.0 %. The quiet check of the speed run (one sample about every 10 seconds) took 52 samples, the one of the supplementary run 39 samples and the one of the run at the window of 32,768 tokens 5 samples. All of them show only the kernel's memory thread kswapd0, at most 1.8 % of a core. The quiet-check tables are not published.

python3 speed2.py --base-url http://127.0.0.1:45917/v1 --label tf5090-rel126-1111-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/tf5090-rel126-1111-speed-r0/tf5090-rel126-1111-speed-r0/speed2 --contract $RUN/tf5090-rel126-1111-speed-r0/tf5090-rel126-1111-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/engines/bob-2410/models/Qwen3.8-27B-MLX-4bit/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode) --model qwen38-27b --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source BobClawblaw/qwen38-27b-vision-rtx-5090@2147ce9d README.md:277 (PARALLEL auto = streams at once as many as the window allows: one at 262,144; more streams were not measured on this engine); serve.sh:28 (--parallel auto) --server-ctx 262144 --server-ctx-source card box-6-tensorfold-1.2.6-q35-int8-mlx4bit-dflash2-5090 field ctx

# Long prompts: supplementary run of 2026-11-11 (raw folder tf5090-rel126-1111-ctx-speed-r0-ctx)
python3 speed2.py --base-url http://127.0.0.1:33609/v1 --label tf5090-rel126-1111-ctx-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/tf5090-rel126-1111-ctx-speed-r0/tf5090-rel126-1111-ctx-speed-r0/speed2 --contract $RUN/tf5090-rel126-1111-ctx-speed-r0/tf5090-rel126-1111-ctx-speed-r0/speed2/speed-contract.json --parts ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/engines/bob-2410/models/Qwen3.8-27B-MLX-4bit/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode) --model qwen38-27b --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source BobClawblaw/qwen38-27b-vision-rtx-5090@2147ce9d README.md:277 (PARALLEL auto = streams at once as many as the window allows: one at 262,144; more streams were not measured on this engine); serve.sh:28 (--parallel auto) --server-ctx 262144 --server-ctx-source card box-6-tensorfold-1.2.6-q35-int8-mlx4bit-dflash2-5090 field ctx --supplement-of tf5090-rel126-1111

# Requests at once, a deviation from the recipe (window of 32768 tokens): supplementary run of 2026-11-11 (raw folder tf5090-rel126-1111-conc-speed-r0-conc)
python3 speed2.py --base-url http://127.0.0.1:39041/v1 --label tf5090-rel126-1111-conc-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/tf5090-rel126-1111-conc-speed-r0/tf5090-rel126-1111-conc-speed-r0/speed2 --contract $RUN/tf5090-rel126-1111-conc-speed-r0/tf5090-rel126-1111-conc-speed-r0/speed2/speed-contract.json --parts conc --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/engines/bob-2410/models/Qwen3.8-27B-MLX-4bit/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode) --model qwen38-27b --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 6 --server-slots-source DEVIATION window 32768 (CONTEXT/Q2_CTX of Bob's serve.sh, --parallel auto): the author's window 262144 gives ONE stream, so C2/C4/C8 would only queue; at 32768 the server itself chooses its streams, read from the startup line of the start test Nov 11, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 17.67 GiB; 6 streams at once, 1.98 GiB each at a 32768-token window (fp8 caches), of 12.2 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' --server-ctx 32768 --server-ctx-source DEVIATION from Bob's CONTEXT default 262144 (reason: real parallel lanes for the concurrency levels; 32768 is also the engine default for Qwen3.8-27B per TensorFold v1.2.6 README.md:307; the main run tf5090-rel126-1111 keeps the author's window); --supplement-of tf5090-rel126-1111

How well it answers

All seven Quality v2 tests were run with the window of 262,144 tokens, one question at a time (the stream count 1 of the recipe). Thinking was off in six of them; GPQA 30 ran with thinking on.

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-11.

Note from the run: MMLU-Pro and GPQA Diamond ran with an answer limit of 48 tokens: the tool asks for the letter only, and the median reply has 2 tokens. In GPQA Diamond one reply (question gpqa-m161, the text '2.25') had no letter the tool could read and counts as wrong. IFEval ran with 4,096 tokens (prompt-level strict), LiveCodeBench with 32,768, long text with 128 and GPQA 30 with thinking with 16,384. Cut off at their limits: 0 of 400 MMLU-Pro, 0 of 198 GPQA and 0 of 200 IFEval answers, 1 of 40 LiveCodeBench answers (at 32,768 tokens) and none in the long-text tests; 10 of the 30 thinking answers were cut off at the limit. A cut-off answer counts as wrong. All questions of a test ran one at a time, and the time per question of the thinking test is the time of a single question while no other request was running on the server: it is not a speed value. GPQA Diamond ran from the public mirror fingertap/GPQA-Diamond (198 questions), because the original is gated and no Hugging Face token was available.

LiveCodeBench: the 40 newest tasks of the set (contest dates 2025-03-15 to 2025-04-06), pass@1, the code runs in a sandbox without network. Of the 20 answers that failed, 1 was cut off at the limit and counts as wrong, and 7 ended with a stop and had no code in the reply (the tool counts them as failed; the replies are between 166 and 9,946 tokens long, counted from the answer file, which the site does not publish). The other 12 had code that failed the tests. By difficulty, 7 of 9 easy, 7 of 11 medium and 6 of 20 hard tasks passed (also counted from the answer file). Long text at 8K and 32K, 4 tasks of 10 each, scored strictly: multi-key retrieval 10 of 10 and 10 of 10, variable tracking 6 of 10 and 5 of 10, common-word extraction 10 of 10 and 9 of 10, multi-hop question answering 8 of 10 and 4 of 10. A variable-tracking task counts only when all variables are named (mean fraction 0.7 at 8K and 0.65 at 32K), and a multi-hop answer counts when the expected answer, normalised (lower case, no punctuation or articles), appears in the reply as a run of words. The 32K prompts have 32,779 and 32,780 tokens. Four of the 80 prompts are one token away from the exact length (8,191, 8,193, 32,767 and 32,767 tokens; the tool marks them not exact).

Thinking test (GPQA 30, thinking on): 30 of the GPQA questions, with the model card's thinking setting (temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence penalty 0; model card line 252), seed 42, a limit of 16,384 tokens. 10 of the 30 were still thinking at the limit (every one of them at exactly 16,384 tokens) and count as wrong. Of the 20 answers that finished, 19 are right. For agentic tasks the Qwen model card suggests a maximum output length of 262,144 tokens for the reasoning and 131,072 for the final response (lines 507 to 510), so 16,384 tokens is far below that, and 63.3 % is the score at this limit and not at the card's length. The model card lists a presence penalty of 0 for the thinking setting (line 252), which is what this test used. The median of 9,178 tokens and 32.8 seconds per question covers all 30 questions, including the 10 at the limit. The answer file of this test is not published: the site's filter for power figures holds it back because model text in it matches the filter. The summary (counts, tokens, seconds), the log and the contract are published.

Sampling. The tests with thinking off sent the model card's setting for replies without thinking (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5; model card line 253) with the request flag for no thinking; every request of the quality tests sent cache_prompt false and seed 42. One run of each test, no repetition.

Foreign load. These tests have no load guard; their quiet check lists the busiest other process about every 10 seconds. MMLU-Pro, GPQA Diamond, IFEval, LiveCodeBench and long text (one run, 189 samples): 2 samples are above 25 % of a core. One is 99.8 % at 15:05:57 for '/usr/local/bin/python /work/solution.py', the code sandbox of the LiveCodeBench test itself (a Python process running the solution under test; the tool starts it in a docker container without network), and one is 33.3 % at 15:15:38 for a process named '[runc:[2:INIT]]' while a long-text question was running. I did not find out what started that process, and nothing was repeated. The other 187 samples show only kswapd0, at most 1.8 %. GPQA 30 with thinking: 102 samples, all of them kswapd0, at most 1.8 %. The card was empty (2 MiB in use) before the server started. Not published: the LiveCodeBench answer file and its code and test folders (the site's filter holds back source files), the answer file of the thinking test (see above), the quiet-check tables and the prompt files of the long-text tests; the summaries, logs and contracts of all tests and the answer files of the other tests are published.

First attempt, interrupted. The quality tests were started twice. The first attempt (raw folder tf5090-rel126-1111-inst-r2, started at 14:34 machine time) broke off during its first test, MMLU-Pro. Its log ends at question 200 of 400 (14:35:15). The answer file holds 202 answered questions and then 38 error lines: question 203 got 'RemoteDisconnected: Remote end closed connection without response' and every later one a refused connection (URLError, Errno 111), so the server no longer answered. The GPQA file holds 20 error lines, all refused connections, and the error files of both tests end with 'BrokenPipeError: [Errno 32] Broken pipe', raised when the tool wrote a progress line. The files do not show what made that pipe break, and I did not find out. Nothing from this attempt is in any number of this page, and the job's register has no row for it. The same five tests were started again on a fresh server at 14:45 and ran to the end, one run each: they are the results above. I publish the files of the first attempt so that the break-off can be checked.

Raw files of the first attempt: log · error file (MMLU-Pro) · error file (GPQA) · MMLU-Pro answers · GPQA answers.

bash tools/quality2/measure_all.sh tools/quality2/cards/tensorfold-1.2.6-q35-int8-mlx4bit-dflash2-5090.json --parts mmlupro --job tf5090-rel126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/tensorfold-1.2.6-q35-int8-mlx4bit-dflash2-5090.json --parts gpqa --job tf5090-rel126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/tensorfold-1.2.6-q35-int8-mlx4bit-dflash2-5090.json --parts ifeval --job tf5090-rel126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/tensorfold-1.2.6-q35-int8-mlx4bit-dflash2-5090.json --parts lcb --job tf5090-rel126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/tensorfold-1.2.6-q35-int8-mlx4bit-dflash2-5090.json --parts ruler --job tf5090-rel126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/tensorfold-1.2.6-q35-int8-mlx4bit-dflash2-5090.json --parts ruler --job tf5090-rel126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/tensorfold-1.2.6-q35-int8-mlx4bit-dflash2-5090.json --parts think --job tf5090-rel126-1111

How much context fits

Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed at the largest size was not measured. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. Measured 2026-11-11. Raw files: result of every step · log · contract.

Note from the run: The largest size tested is 262,144 tokens (256K), the top step of the series, the window of the recipe and of the speed and quality tests, and the model's own window (Qwen model card line 53). It loaded and answered with a prompt of 249,036 tokens (95 % of the window). I did not test sizes above it. How this differs from the speed and quality tests: the memory test starts the server once per size with the recipe's serve.sh and the window set to that size (our start script passes the size on as the recipe's own CONTEXT variable). At 262,144 tokens it is the same command as the server of the speed and quality tests. The seven server logs of this test are not published: they hold only lines of our start script, not the server's start text, so the number of streams at the smaller windows is not shown. Peak memory is the memory in use on the whole card (nvidia-smi; it was empty before each start, 2 MiB). It rises from 23.61 GiB at 4K to 28.75 GiB at 64K and is lower at the larger windows: 27.76 GiB at 128K and 26.99 GiB at 256K. I did not log the number of streams or the cache size at the smaller windows, so I cannot say why the peak does not rise with the window. Decode speed at these sizes is not measured (the server returns no timings).

bash tools/quality2/measure_all.sh tools/quality2/cards/tensorfold-1.2.6-q35-int8-mlx4bit-dflash2-5090.json --parts fit --job tf5090-rel126-1111

Also made this possible

• Engine (TensorFold official release v1.2.6, 10 Nov 2026, 12:51 UTC)Ash Hart@ashxhart GitHub Hugging Face Website
• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• Model files (MLX 4-bit, revision 22d8d538); Hugging Face organisation TensorFold, whose only listed member is ashxhartTensorFoldHugging Face @ashxhart GitHub Website
• Draft model (DFlash2); the revision loaded is z-lab's mirror 50307d4c, whose model card says it mirrors this repositoryInco AI@inco_ai Hugging Face GitHub Website
• DFlash method and the mirror of the drafter that was loaded (revision 50307d4c); the mirror's model card asks to cite the original DFlash paper and links github.com/z-lab/dflashz-lab@zhijianliu_ Hugging Face GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one GPU, the whole model in video memory). Driver 610.43.02, kernel 6.12.95+deb13-amd64

Model

TensorFold/Qwen3.8-27B-MLX-4bit @22d8d538 + z-lab/Qwen3.8-27B-DFlash2 @50307d4c

Model files (MLX 4-bit, revision 22d8d538); Hugging Face organisation TensorFold, whose only listed member is ashxhart

huggingface.co/TensorFold/Qwen3.8-27B-MLX-4bit

Draft model (DFlash2); the revision loaded is z-lab's mirror 50307d4c, whose model card says it mirrors this repository

huggingface.co/incoai/Qwen3.8-27B-DFlash2

DFlash method and the mirror of the drafter that was loaded (revision 50307d4c); the mirror's model card asks to cite the original DFlash paper and links github.com/z-lab/dflash

huggingface.co/z-lab/Qwen3.8-27B-DFlash2

Engine

1x NVIDIA GeForce RTX 5090 32 GB (one GPU, the whole model in video memory). Driver 610.43.02, kernel 6.12.95+deb13-amd64. The recipe's serve.sh as written: window of 262,144 tokens, 1 stream, fp8 attention cache, int8 prompt matmuls, vision option on, with the binary of the official release archive — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.6

Recipe (repository qwen38-27b-vision-rtx-5090, revision 2147ce9d), and the Qwen3.8-27B CUDA engine code that the release changelog credits to him (pull requests #1079, #1080, #1168 and #1176)

github.com/BobClawblaw/qwen38-27b-vision-rtx-5090 by BobClawblaw

Steps

1. Get the recipe, the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   tensorfold-native serve $MODELS/Qwen3.8-27B-MLX-4bit \
     --drafter $MODELS/Qwen3.8-27B-DFlash2 \
     --name qwen38-27b \
     --alias qwen38-nvfp4 \
     --host 127.0.0.1 \
     --port 8888 \
     --context 262144 \
     --parallel auto \
     --kv-cache fp8 \
     --prompt-precision int8 \
     --vision

   What the parts mean

   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on
   • --drafter $MODELS/Qwen3.8-27B-DFlash2the draft model: it guesses the next tokens and the main model checks them; without it the model decodes plain
   • --name qwen38-27bthe model ID the server advertises to clients
   • --alias qwen38-nvfp4the model name clients use in their requests

TensorFold 1.2.6 tensorfold-native (Zig, CUDA, sm_120), official release archive tensorfold-1.2.6-linux-x86_64, DFlash2 drafts, int8 prompts, vision on; tensorfold-native serve $MODELS/Qwen3.8-27B-MLX-4bit --drafter $MODELS/Qwen3.8-27B-DFlash2 --name qwen38-27b --alias qwen38-nvfp4 --host 127.0.0.1 --port 8888 --context 262144 --parallel auto --kv-cache fp8 --prompt-precision int8 --vision (recipe: BobClawblaw/qwen38-27b-vision-rtx-5090 revision 2147ce9d, serve.sh lines 23 to 30, started as ./serve.sh with PORT=8888 HOST=127.0.0.1; serve.sh sets TENSORFOLD_CUDA_KERNELS=native itself, line 19; $MODELS is the folder with the two checkpoints and $TF_SRC the folder with the unpacked release archive)