Mia's Qwen3.8-27B EXL3 3.5 bpw pack on TensorFold 1.2.7 and one RTX 5090 (DFlash2 drafts, window 131,072 tokens)

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds; long prompts (later supplementary run of 2026-11-12, same setup): prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer · speed result · speed contract · log.

What ran. The speed test ran TensorFold 1.2.7 as the official release archive for x86_64 Linux (tensorfold-1.2.7-linux-x86_64.tar.gz, published 11 Nov 2026, 17:18 UTC by the repository owner, sha256 319f548b, the digest GitHub lists for that file; the binary tensorfold-native inside has the sha256 8e5729ba that the measuring machine recorded before the runs; no patch, no container). The command is the docs command for EXL3 checkpoints (docs/recipes/qwen3.8-27b.md lines 279 to 283: tensorfold serve <pack folder> --host 0.0.0.0 --port 8080, there with turboderp's 3.00bpw pack), started with the binary tensorfold-native of that archive (README line 79 defines tensorfold as a call of tensorfold-native), with Mia's pack in place of turboderp's. The pack is Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw at revision 19441ac8 and the drafter is z-lab/Qwen3.8-27B-DFlash2 at revision 50307d4c; the marker lines of the file check (17 files of the pack, the drafter, the release) are in the start text recorded with the run. I added --drafter z-lab/Qwen3.8-27B-DFlash2, because the README says that Qwen3.8-27B uses DFlash2 through that flag (line 323), a name, and a local host and port and, for this page, --context 131072 (see Window and streams). Our own combination. The docs of this release do not name Mia's pack for TensorFold: docs/recipes/qwen3.8-27b.md lines 233 to 235 list her packs and say that none has been loaded there yet, docs/recipes/cuda.md lines 92 to 94 list the 3.5bpw pack under 'Not tried yet', and the only EXL3 pack measured there is turboderp's 3.00bpw (lines 304 and 305). So this page is a combination of Mia's pack with Ash's engine and the docs' EXL3 command, measured by me; it is not a published recipe. I show no number of the docs next to ours. Window and streams. The window is 131,072 tokens, set with --context 131072; the server's default is 32,768 tokens (README line 307) and the model's own window is 262,144 tokens. My tests need more than 32,768 tokens in one window: LiveCodeBench answers of up to 32,768 tokens, the 32K long-text prompts (32,779 and 32,780 tokens) and the long-prompt steps from 32K. The README says that --context bounds prompt plus reply (line 307). This is my deviation from the command, and every number on this page was measured with it. With this window the server chose the streams itself (--parallel auto asks for up to eight streams as memory and the engine allow, README lines 104 and 345): its start line reads '1 stream at once, 8.96 GiB each at a 131072-token window, of 14.7 GiB left after a 1.0 GiB reserve' (the model takes 15.20 GiB with the drafter), and GET /v1/models reports a context of 131072 (start line and speed result, recorded with the run). Long prompts. The 128K step could not use 131,072 prompt tokens: with 512 reply tokens that is 131,584 tokens against the window of 131,072. The test then takes the window minus 1,024, which is 130,048 tokens, as its rule says (valid from 126,976). The row for 128K in the table of long prompts was therefore measured at 130,048 tokens; the other three steps (16K, 32K, 64K) have the exact length. The prefill series of this card has two sizes, 2K and 8K, so the tiles for prefill and first token stop at 8K; the values for 16K, 32K, 64K and 128K are in the table of long prompts. The long-prompt steps were measured twice. The speed run itself contained them (raw folder mia27exl3-35-5090-127-ctx131072-1111-speed-r0), and a supplementary run of the same day repeated only these steps (raw folder mia27exl3-35-5090-127-ctx131072-1111-ctx-speed-r0-ctx, same command, the server started anew). The table shows the supplementary run. The values of the speed run are: 16K: 214.1 tok/s (code) and 161.3 tok/s (prose), prefill 2034.8 tok/s, first token after 8.1 s; 32K: 157.9 tok/s (code) and 153.5 tok/s (prose), prefill 1955.2 tok/s, first token after 16.8 s; 64K: 148.8 tok/s (code) and 138.7 tok/s (prose), prefill 1819.0 tok/s, first token after 36.0 s; 128K step (130,048 tokens): 138.8 tok/s (code) and 134.4 tok/s (prose), prefill 1611.9 tok/s, first token after 80.7 s. Requests at once. The server has 1 stream. '1 at once' is inside it; '2', '4' and '8 at once' send 2, 4 and 8 requests to 1 stream (the others wait), and the tool logs them as measured anyway. So these rows measure the waiting requests as well: '8 at once' does not mean 8 streams, and the rows for 2, 4 and 8 at once are not comparable with the row for 1 at once. The total rises from 150.1 tok/s with 1 request to 151.1 tok/s with 8 requests, while the speed of one request falls from 150.1 to 34.4 tok/s. The headline 200.7 tok/s is the mean of the code prompt (209.9) and the prose prompt (191.4), medians of two runs each (spread (CV) of 9.4 % and 0.4 %). The code prompt has 1,258 and 1,260 tokens and the prose prompt 1,351 and 1,355 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 21.0 % (1,544 of 7,344 drafted tokens, mean accepted length 4.1 tokens). The long-answer test asks for one reply of up to 32,768 tokens; the reply had 5,104 tokens and ended with a stop after 19.6 seconds. Sampling is the Qwen model card's setting for replies without thinking (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5), sent with the request flag for no thinking and seed 42; the contracts of the speed runs name the source. The chat template thinks by default and the command passes no sampling flag, so the server's own default is temperature 1.0, top_p 0.95, top_k 20 (start line). Foreign load. The load guard (any other process above 25 % of a core for two samples in a row, sampled twice a second) was set to ignore one kernel thread, kcompactd0 (memory compaction). In all 35 steps of the speed run and all 16 steps of the supplementary run it found no other process above its limit, so no step was repeated; the highest other process in any step was the Docker daemon (dockerd) at 18.0 % of a core. The quiet check of the speed run (one sample about every 10 seconds) took 79 samples and the one of the supplementary run 64 samples; all of them show only the kernel's memory thread kswapd0, at most 1.7 % of a core.

python3 speed2.py --base-url http://127.0.0.1:36873/v1 --label mia27exl3-35-5090-127-ctx131072-1111-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/mia27exl3-35-5090-127-ctx131072-1111-speed-r0/mia27exl3-35-5090-127-ctx131072-1111-speed-r0/speed2 --contract $RUN/mia27exl3-35-5090-127-ctx131072-1111-speed-r0/mia27exl3-35-5090-127-ctx131072-1111-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/mia-exl3-35bpw-1111/model/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source THINKING: Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw README.md:75-77 (revision 19441ac8; local copy $HOME/mia-exl3-35bpw-1111/model/README.md lines 75-77): 'Recommended sampling (Qwen3.8 thinking mode): temperature 0.6, top_p 0.95, top_k 20.' min_p 0.0 and presence_penalty 0.0 are not named by Mia, they are the Qwen card's thinking-mode values for the parameters left open (Qwen/Qwen3.8-27B README.md line 252). INSTRUCT (non-thinking): Mia's card names nothing for it, the Qwen card applies: https://huggingface.co/Qwen/Qwen3.8-27B README.md line 253 (temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5), local copy not published line 252-253. The engine default of the docs command is the checkpoint's generation_config.json (server line: sampling temperature 1.0, top_k 20, top_p 0.95) --model qwen38-27b-exl3-35 --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source TensorFold v1.2.7 README.md:104 and :345 (--parallel auto, the default the docs command uses, asks for up to eight streams as memory and the engine allow); streams = the server's own choice, read from the startup line at the probe start Nov 11, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 15.20 GiB; 1 stream at once, 8.96 GiB each at a 131072-token window, of 14.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' --server-ctx 131072 --server-ctx-source DEVIATION named: --context 131072 (see _comment). Startup line of the probe Nov 11, 2026 (release 1.2.7, RTX 5090, with --drafter): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 15.20 GiB; 1 stream at once, 8.96 GiB each at a 131072-token window, of 14.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' and '[tensorfold] serving qwen38-27b-exl3-35 at http://127.0.0.1:8133/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; drafts: DFlash2 (z-lab/Qwen3.8-27B-DFlash2) + copies; context: 131072; loaded in 20.2s)'. Window source: README.md:307 (v1.2.7), row of --context N: “Bound prompt plus reply tokens; the model's window and memory checks still apply. Qwen3.8-27B defaults to 32,768 and takes up to 262,144.”; the docs command sets no context.

# Long prompts: supplementary run of 2026-11-12 (raw folder mia27exl3-35-5090-127-ctx131072-1111-ctx-speed-r0-ctx)
python3 speed2.py --base-url http://127.0.0.1:37401/v1 --label mia27exl3-35-5090-127-ctx131072-1111-ctx-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/mia27exl3-35-5090-127-ctx131072-1111-ctx-speed-r0/mia27exl3-35-5090-127-ctx131072-1111-ctx-speed-r0/speed2 --contract $RUN/mia27exl3-35-5090-127-ctx131072-1111-ctx-speed-r0/mia27exl3-35-5090-127-ctx131072-1111-ctx-speed-r0/speed2/speed-contract.json --parts ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/mia-exl3-35bpw-1111/model/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source THINKING: Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw README.md:75-77 (revision 19441ac8; local copy $HOME/mia-exl3-35bpw-1111/model/README.md lines 75-77): 'Recommended sampling (Qwen3.8 thinking mode): temperature 0.6, top_p 0.95, top_k 20.' min_p 0.0 and presence_penalty 0.0 are not named by Mia, they are the Qwen card's thinking-mode values for the parameters left open (Qwen/Qwen3.8-27B README.md line 252). INSTRUCT (non-thinking): Mia's card names nothing for it, the Qwen card applies: https://huggingface.co/Qwen/Qwen3.8-27B README.md line 253 (temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5), local copy not published line 252-253. The engine default of the docs command is the checkpoint's generation_config.json (server line: sampling temperature 1.0, top_k 20, top_p 0.95) --model qwen38-27b-exl3-35 --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source TensorFold v1.2.7 README.md:104 and :345 (--parallel auto, the default the docs command uses, asks for up to eight streams as memory and the engine allow); streams = the server's own choice, read from the startup line at the probe start Nov 11, 2026: '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 15.20 GiB; 1 stream at once, 8.96 GiB each at a 131072-token window, of 14.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' --server-ctx 131072 --server-ctx-source DEVIATION named: --context 131072 (see _comment). Startup line of the probe Nov 11, 2026 (release 1.2.7, RTX 5090, with --drafter): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 15.20 GiB; 1 stream at once, 8.96 GiB each at a 131072-token window, of 14.7 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' and '[tensorfold] serving qwen38-27b-exl3-35 at http://127.0.0.1:8133/v1 (sampling: temperature 1.0, top_k 20, top_p 0.95; drafts: DFlash2 (z-lab/Qwen3.8-27B-DFlash2) + copies; context: 131072; loaded in 20.2s)'. Window source: README.md:307 (v1.2.7), row of --context N: “Bound prompt plus reply tokens; the model's window and memory checks still apply. Qwen3.8-27B defaults to 32,768 and takes up to 262,144.”; the docs command sets no context. --supplement-of mia27exl3-35-5090-127-ctx131072-1111

How well it answers

Three of the seven Quality v2 tests were run with the window of 131,072 tokens and 1 request at a time, the 1 stream of the server: LiveCodeBench and long text at 8K and 32K. Thinking was off in all of them. The other four tests (MMLU-Pro, GPQA Diamond, IFEval and the thinking test) and the memory test are on the page of the same setup with the server's default window.

Thinking off in all three tests. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-12.

Note from the run: Three of the seven Quality v2 tests were run with the window of 131,072 tokens: LiveCodeBench and long text at 8K and 32K. LiveCodeBench ran with an answer limit of 32,768 tokens, long text with 128, both with thinking off and 1 request at a time (the server has 1 stream). A cut-off answer counts as wrong. The other four tests (MMLU-Pro, GPQA Diamond, IFEval and the thinking test) are on the page of the same setup with the server's default window.

LiveCodeBench: the 40 newest tasks of the set (contest dates 2025-03-15 to 2025-04-06), pass@1, the code runs in a sandbox without network. 24 of 40 passed. Of the 16 that failed, 1 was cut off at 32,768 tokens and counts as wrong. By difficulty, 9 of 9 easy, 10 of 11 medium and 5 of 20 hard tasks passed (counted from the answer file, which the site does not publish).

Long text at 8K and 32K, 4 tasks of 10 each, scored strictly: multi-key retrieval 10 of 10 and 10 of 10, variable tracking 6 of 10 and 1 of 10, common-word extraction 10 of 10 and 9 of 10, multi-hop question answering 8 of 10 and 3 of 10. A variable-tracking task counts only when all variables are named (mean fraction 0.7 at 8K and 0.425 at 32K), and a multi-hop answer counts when the expected answer, normalised (lower case, no punctuation or articles), appears in the reply as a run of words. The 32K prompts have 32,779 and 32,780 tokens. Three of the 80 prompts are one token away from the exact length (8,191, 8,193 and 32,767 tokens; the tool marks them not exact).

Sampling. Both tests sent the Qwen model card's setting for replies without thinking (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5) with the request flag for no thinking; every request sent cache_prompt false and seed 42. One run of each test, no repetition.

Foreign load. These tests have no load guard; their quiet check lists the busiest other process about every 10 seconds. LiveCodeBench and long text: 186 samples, 4 of them above 25 % of a core (97.8, 60.0, 33.3, 28.5 %: 2 of them the code sandbox of the LiveCodeBench test itself, a Python process running the solution under test, and 2 a Python one-liner that opens a local web address (I did not trace which program starts it)). The other 182 samples are below 25 % of a core, at most 19.3 %. Nothing was repeated. The card was empty (2 MiB in use) before the server started. Not published: the LiveCodeBench answer file and its code and test folders (the site's filter holds back source files), the quiet-check tables, and the prompt files of the long-text tests; the summaries, logs and contracts of both tests and the answer file of the long-text test are published.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-3.5bpw-mia-pack-tensorfold-1.2.7-5090-ctx131072-1111.json --parts lcb --job mia27exl3-35-5090-127-ctx131072-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-3.5bpw-mia-pack-tensorfold-1.2.7-5090-ctx131072-1111.json --parts ruler --job mia27exl3-35-5090-127-ctx131072-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-exl3-3.5bpw-mia-pack-tensorfold-1.2.7-5090-ctx131072-1111.json --parts ruler --job mia27exl3-35-5090-127-ctx131072-1111

Also made this possible

• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• EXL3 format and quantisation tool (exllamav3)turboderp@turboderp_ GitHub Hugging Face
• Engine (TensorFold official release v1.2.7, 11 Nov 2026, 17:18 UTC)Ash Hart@ashxhart GitHub Hugging Face Website
• Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)Inco AI@inco_ai Hugging Face GitHub Website
• Speculative decoding method (DFlash) and the mirror we loadedz-lab@zhijianliu_ GitHub Hugging Face

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one GPU, the whole model in video memory). Host: AMD Ryzen 9 9950X, 96 GB RAM installed (92 GiB usable). Driver 610.43.02, kernel 6.12.95+deb13-amd64

Model

Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw @19441ac8 (EXL3 v1.4.2, 3.5 bpw module-adaptive, head 6 bit, mul1 codebook) + z-lab/Qwen3.8-27B-DFlash2 @50307d4c (drafter); OWN COMPOSITION: Mia's pack on Ash's engine, no TensorFold recipe by Mia, the TensorFold docs list the pack as not loaded yet

Quant pack (EXL3 3.5 bpw, revision 19441ac8) and model card

huggingface.co/Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw

Draft model (DFlash2, revision 50307d4c)

huggingface.co/z-lab/Qwen3.8-27B-DFlash2

Engine

TensorFold 1.2.7, official release for x86_64 Linux (native CUDA, sm_120), Mia's EXL3 pack with DFlash2 drafts, window 131,072 tokens, 1 stream chosen by the server — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.7

Context window

131,072 tokens: how much text the model can see at once

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8133, only from this computer.

   Copy

   tensorfold-native serve model \
     --drafter z-lab/Qwen3.8-27B-DFlash2 \
     --name qwen38-27b-exl3-35 \
     --host 127.0.0.1 \
     --port 8133 \
     --context 131072

   The file names are the ones we used. Run the command in the folder that holds them, or put the full path in front of each name.

   What the parts mean

   • --host 127.0.0.1who can connect: only this computer
   • --port 8133the network port the server listens on
   • --drafter z-lab/Qwen3.8-27B-DFlash2the draft model: it guesses the next tokens and the main model checks them; without it the model decodes plain
   • --name qwen38-27b-exl3-35the model ID the server advertises to clients

TensorFold native (Zig) 1.2.7 official release archive tensorfold-1.2.7-linux-x86_64.tar.gz (sha256 319f548b), CUDA (chip nvidia-sm120), DFlash2 drafts (named deviation --drafter), window 131072 (named deviation --context 131072), 1x RTX 5090 (sm_120); tensorfold-native serve $HOME/model --drafter z-lab/Qwen3.8-27B-DFlash2 --name qwen38-27b-exl3-35 --host 127.0.0.1 --port 8133 --context 131072 (docs command: tensorfold serve <pack folder> --host 0.0.0.0 --port 8080, docs/recipes/qwen3.8-27b.md:279-283, EXL3 checkpoints, there with turboderp's 3.00bpw pack and tensorfold pull z-lab/Qwen3.8-27B-DFlash2; I used Mia's pack in the folder model (revision 19441ac8, 17 files) and added --drafter (README.md:323), --name, a local host and port and --context 131072 (the default is 32,768, README.md:307))