Qwen3.8-Flash-Next on one RTX 5090 with a 131,072-token window: TensorFold from a pull request head with own patches (our patch 3, four patches, 1 stream)
Weights: EXL3 3.05 bpw (quantised by turboderp, branch 3.05bpw_h5_ng5, revision 69e33439)
1x NVIDIA GeForce RTX 5090 32 GB (one GPU); the model's experts sit in pinned host memory. Host: AMD Ryzen 9 9950X, 96 GB RAM installed, PCIe 5.0 x16. Driver 610.43.02, CUDA 13.3 (nvcc 13.3 for the build), kernel 6.12.95+deb13-amd64
Not ranked yet. The rank needs a finished, verified Gauntlet race (the game test), and this setup has not played it yet. It shows Speed v2, Quality v2 and the memory test, every number with its raw files and exact command. How the game test works.
How fast it runs
| Measurement | Result |
|---|---|
| Decode | 147.8 tok/s |
| Words per second | 98.6 words/s |
| Characters per second | 690.1 chars/s |
| Decode, code prompt | 157.7 tok/s |
| Decode, prose prompt | 138.0 tok/s |
| Spread of the decode runs (CV) | 1.6 % |
| Spread of the code runs (CV) | 1.6 % |
| Spread of the prose runs (CV) | 1.5 % |
| Decode, window 1 | 144.2 tok/s |
| Prefill at 2K | 1657.8 tok/s |
| Prefill at 8K | 3521.8 tok/s |
| Prefill at 32K | 3598.3 tok/s |
| Time to first token at 2K | 1.2 s |
| Time to first token at 8K | 2.3 s |
| Time to first token at 32K | 9.1 s |
| Draft acceptance rate | 54.5 % |
| Mean accepted length | 2.5 tokens |
| 1 request at once (total) | 108.0 tok/s |
| 2 at once (total) | 107.3 tok/s |
| 4 at once (total) | 108.4 tok/s |
| 8 at once (total) | 108.6 tok/s |
| 1 request at once (per request) | 108.0 tok/s |
| 2 at once (per request) | 81.2 tok/s |
| 4 at once (per request) | 45.1 tok/s |
| 8 at once (per request) | 24.3 tok/s |
| Server slots (--parallel 1 set by us; the start line confirms 1 stream) | 1 slot |
| Long answer, wall time | 41.8 s |
Long prompts
| Prompt length | Prefill | Time to first token | Decode, code prompt | Decode, prose prompt |
|---|---|---|---|---|
| 16K tokens | 3582.2 tok/s | 4.6 s | 154.3 tok/s | 134.7 tok/s |
| 32K tokens | 3576.9 tok/s | 9.2 s | 156.9 tok/s | 137.8 tok/s |
| 64K tokens | 3564.5 tok/s | 18.4 s | 155.1 tok/s | 145.6 tok/s |
| 128K tokens | 3471.9 tok/s | 37.5 s | 157.8 tok/s | 143.3 tok/s |
Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-13. Raw files: speed result · speed contract · log · long answer.
What ran. Own patches on an unmerged pull request head, not an official TensorFold release: the head 06bdb53c (commit 06bdb53c8d225661f2129064a97a44b58b80cbad) of BlivionIaG's pull request #1443 (open, not merged; base: Ash's TensorFold main branch at commit 22fb6014), built by us with our own patches on top (sha256 90c51b49, recorded with the run; the binary reports its own version as 1.2.6, the version of the base). The four own patches are P1 (table batch), P2 (LRU eviction), P3 (hit first) and P4 (prefetch of n-gram rows), and the setting LINE_DELTAS of the pull request head (uneven expert lines per layer) is switched on. The patches are ours, not by Ash or BlivionIaG; the patch files are not published. This is exactly the build and the settings of the 32K page of our patch 3 (same head, same pack, same box, 1 stream); only the server window differs. Window and expert lines. The window is 131,072 tokens and the server runs 1 stream. The start line reads '1 stream at once, 4.16 GiB each at a 131072-token window, of 4.3 GiB left after a 1.0 GiB reserve' (model 24.71 GiB on the card). The server chose the expert lines from free card memory: 219 uniform-equivalent lines per layer of 1.76 MiB (18.59 GiB on the card, 24.04 GiB free before), where the 32K page has 253 (21.45 GiB), because the longer window needs more memory per stream. The bank of experts in host memory is 43.21 GiB (copied in 45.4 s); the server loaded in 79.1 s. Nothing was lowered for this run: window and stream count are our choice, the author's start script uses 32,768 tokens and 95 expert lines (page of the author's start settings). Context steps. The server window is 131,072 tokens, so a prompt of 131,072 tokens plus the 512 reply tokens would not fit; the 128K step is measured at 130,048 tokens (the server window minus 1,024). The steps at 16K, 32K, 64K and 128K all ran and fit. With 1 stream the decode at depth stays close to the head values (157.8 tok/s code and 143.3 tok/s prose at 130,048 tokens, against 157.7 and 138.0 at a short prompt); the prefill at 130,048 tokens runs at 3,472 tok/s with a first token after 37.5 s. Setup differences from the pull request author's setup. As on the 32K page: our nvcc is CUDA 13.3 and our driver 610.43.02 (the author's receipts name 13.4.92 and 615.71.09); the speed test sends presence penalty 0 instead of the model card's 1.5 because the CUDA backend of this head refuses any other value (HTTP 400); MTP drafts are on (the author's script starts the first head with --no-drafts); the memory reserve is the server's default of 1.0 GiB. Server log. On the first request the server log shows two CUDA graph capture errors (CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED) after which the engine continues without graphs (the switch was at 0); we did not check whether the 32K runs show the same lines. Other 128K pages of this head. The page of the 131,072-token window without our patches (4 streams, 71 expert lines) is a separate page. We also ran our patches at 128K with 4 streams and 71 expert lines (the line deltas had to be switched off there, because they stop the server start at 71 lines): its decode is lower (91.7 tok/s code, 85.7 tok/s prose), and one cell of the 128K step failed (the second code run ended after 24 tokens), so it is not shown as a page. Requests at once. The server has 1 stream, so the levels 2, 4 and 8 at once are above its slots: the requests queue and run one after another. The total stays near the value of one request (108.0, 107.3, 108.4 and 108.6 tok/s with 1, 2, 4 and 8 requests).
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 speed2.py --base-url http://127.0.0.1:46307/v1 --label fn5090-ourpatch3-ctx131072-par1-1311-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/fn5090-ourpatch3-ctx131072-par1-1311-speed-r0/fn5090-ourpatch3-ctx131072-par1-1311-speed-r0/speed2 --contract $RUN/fn5090-ourpatch3-ctx131072-par1-1311-speed-r0/fn5090-ourpatch3-ctx131072-par1-1311-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json <pack folder>/tokenizer.json --tokenizer-python <not published> --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0} --sampling-source Qwen/Qwen3.8-Flash-Next README.md:343 (Instruct/non-thinking) and :342 (Thinking), in the pack turboderp/Qwen3.8-Flash-Next-exl3 README.md:342-343 and :586-587 identical; generation_config.json (1.0/20/0.95) = thinking default, not used for instruct. Seed 42 for speed/quality. Deviation instruct: presence_penalty 0 instead of the model card's 1.5, because the CUDA backend of this pull request head refuses it (HTTP 400 'presence_penalty and frequency_penalty are not supported by backend cuda: send 0 or leave them out', attempt Nov 12, 2026 15:15, nothing measured). --model qwen3.8-flash-next-exl3-3.05bpw --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source our setting --parallel 1 (start wrapper, FN_PARALLEL=1, as ourpatch3 at 32K); the server start line is in the raw folder --server-ctx 131072 --server-ctx-source start argument --context 131072 (FN_CTX) in the start wrapper, as the card of the unpatched 128K run (4 streams); the server start line is in the raw folder (server-A.log). The author's script tools/exl3_serve.sh @5f089bdf:14 sets 32768: 131072 is an increase by us (the model's window is 262144), not a reduction.Quality
Not run on this setup: no Quality v2 test, so the page shows no quality number.
Also made this possible
- Pull request chain that this build comes from (EXL3 on one GPU, experts in host memory, deep prefill; head 06bdb53c of pull request #1443, open, not merged; the own patches on top are ours)BBlivionIaGGitHub
- Model
Qwen@Alibaba_Qwen Hugging Face GitHub Website - EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)
turboderp@turboderp_ Hugging Face GitHub - Engine (TensorFold; the pull request builds on its main branch)
Ash Hart@ashxhart GitHub Hugging Face Website
Run it yourself
What you need and how to start it, step by step. The results below come from exactly this setup.
What you need
- Hardware
- 1x NVIDIA GeForce RTX 5090 32 GB (one GPU); the model's experts sit in pinned host memory. Host: AMD Ryzen 9 9950X, 96 GB RAM installed, PCIe 5.0 x16. Driver 610.43.02, CUDA 13.3 (nvcc 13.3 for the build), kernel 6.12.95+deb13-amd64
- Model
- Qwen/Qwen3.8-Flash-Next, EXL3 3,05 bpw (turboderp/Qwen3.8-Flash-Next-exl3 @69e33439, Branch 3.05bpw_h5_ng5), MTP head
- EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)
- huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3
- Engine
- the same build as the 32K page of our patch 3, now with the 128K window and the context steps up to 128K — the program that loads the model and answers requests · github.com/ashhart/TensorFold
Steps
Get the engine and the model files from the list above.
Start the server with this command. It loads the model into memory.
env CUDA_VISIBLE_DEVICES=0 \ TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> \ TF_FLASHNEXT_EXL3_GRAPH=0 \ TF_FLASHNEXT_TABLE_BATCH=1 \ TF_FLASHNEXT_EVICT=lru \ TF_FLASHNEXT_EXL3_HIT_FIRST=1 \ TF_FLASHNEXT_PLE_PREFETCH=1 \ TF_FLASHNEXT_EXL3_LINE_DELTAS=0:+178,1:+163,2:+128,4:+27,5:+56,6:+98,7:+90,8:+7,9:-26,10:+48,11:-17,12:+36,13:+50,14:-1,15:-129,16:-117,17:-17,18:+68,19:-15,20:-17,21:+43,22:+74,23:-39,24:+4,25:-37,26:+12,27:-27,28:+42,29:+20,30:-25,31:-138,32:-113,33:-23,34:+36,35:-30,36:-40,37:-5,38:+45,39:-101,40:-92,41:-58,42:-10,43:-56,44:-31,45:-37,46:-21,47:-3 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> \ --backend cuda \ --name qwen3.8-flash-next-exl3-3.05bpw \ --port <port> \ --context 131072 \ --parallel 1 \ --no-update-check \ --prompt-cache-gib 0What the parts mean
CUDA_VISIBLE_DEVICES=0which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints--parallel 1how many requests it answers at the same time--name qwen3.8-flash-next-exl3-3.05bpwthe model ID the server advertises to clients
Start text for this setupfor reproducing↓
You do not need this to run the model. It is the start text of this setup, based on the setup card of this run, with local paths, machine names and addresses shortened or replaced by placeholders, notes added on some pages, and the engine build in front, so anyone can repeat the result.
TensorFold native (Zig), own build from the unmerged pull request head 06bdb53c (BlivionIaG, #1443) plus our own patches (our patch 3: P1 TABLE_BATCH=1, P2 EVICT=lru, P3 HIT_FIRST=1, P4 PLE_PREFETCH=1 reads n-gram rows early into the page cache; LINE_DELTAS uneven expert lines per layer), -Dsm=120, nvcc 13.3; MTP drafts on; engine author Ash (ashhart); the patches are not by Ash or BlivionIaG; env CUDA_VISIBLE_DEVICES=0 TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> TF_FLASHNEXT_EXL3_GRAPH=0 TF_FLASHNEXT_TABLE_BATCH=1 TF_FLASHNEXT_EVICT=lru TF_FLASHNEXT_EXL3_HIT_FIRST=1 TF_FLASHNEXT_PLE_PREFETCH=1 TF_FLASHNEXT_EXL3_LINE_DELTAS=0:+178,1:+163,2:+128,4:+27,5:+56,6:+98,7:+90,8:+7,9:-26,10:+48,11:-17,12:+36,13:+50,14:-1,15:-129,16:-117,17:-17,18:+68,19:-15,20:-17,21:+43,22:+74,23:-39,24:+4,25:-37,26:+12,27:-27,28:+42,29:+20,30:-25,31:-138,32:-113,33:-23,34:+36,35:-30,36:-40,37:-5,38:+45,39:-101,40:-92,41:-58,42:-10,43:-56,44:-31,45:-37,46:-21,47:-3 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> --backend cuda --name qwen3.8-flash-next-exl3-3.05bpw --port <port> --context 131072 --parallel 1 --no-update-check --prompt-cache-gib 0Context and sampling: 131,072 tokens, 1 stream (--parallel 1), expert lines chosen by the server from free card memory (219 uniform-equivalent lines per layer after the line deltas, 18.59 GiB on the card, 24.04 GiB free before), CUDA graph switch 0, MTP drafts on, prompt cache off. Speed test with thinking off per request, sampling of the instruct profile of the model card with presence penalty 0, seed 42.. Hardware: 1x NVIDIA GeForce RTX 5090 32 GB (one GPU); the model's experts sit in pinned host memory. Host: AMD Ryzen 9 9950X, 96 GB RAM installed, PCIe 5.0 x16. Driver 610.43.02, CUDA 13.3 (nvcc 13.3 for the build), kernel 6.12.95+deb13-amd64. All raw files of this setup: raw/v2/qwen38-flash-next-5090-exl3-3-05bpw-tensorfold-pr-06bdb53c-our-patch3-ctx131072-1stream-blivion.