niklaslenzQwen3.8-Flash-Next on one RTX 5090: TensorFold from a pull request head with own patches (more expert lines, 1 stream)
1x NVIDIA GeForce RTX 5090 32 GB (one GPU); the model's experts sit in pinned host memory. Host: AMD Ryzen 9 9950X, 96 GB RAM installed, PCIe 5.0 x16. Driver 610.43.02, CUDA 13.3 (nvcc 13.3 for the build), kernel 6.12.95+deb13-amd64
Not played yet. A game rank needs a finished, verified Gauntlet race. It shows Speed v2, every number with its raw files and exact command. How the game test works.
Own patches on an unmerged pull request head, not an official TensorFold release. The head is 06bdb53c of BlivionIaG's pull request #1443; we built it with our own patches on top, and neither the patches nor this build are part of a TensorFold release. The patch files are not published.
our tuning plus own patches: outputs identical in the equality checks listed in the Speed section below (probes on separate server starts, not register runs). What differs from the page of our tuning with 1 stream (same head, pack, box, window, stream count, total of expert lines and reserve): our patches P1 (table batch), P2 (LRU eviction) and P3 (hit first); LINE_DELTAS is a setting of the pull request head (uneven expert lines per layer), not a patch of ours. All four are switched on by environment variables in the start command; the patch files are not published. The equality checks are probes on separate server starts, not register runs.
Incomplete run. Only Speed v2 was run: no Quality v2 test and no memory test. This setup has no rank and no overall score. The server window of 32,768 tokens (the pull request author's start script) drops these steps, which have no value: the prefill step at 32K, the context steps at 32K, 64K and 128K, and the long-answer step.
The same head has four more pages: the pull request author's start settings (window 32,768 tokens, 1 stream, 95 expert lines), the 131,072-token window (4 streams, expert lines from free memory), our tuning with 1 stream (the same settings without the own patches, window 32,768 tokens, 1 stream) and our tuning with 4 streams (4 streams, window 32,768 tokens).
How fast it runs
| Measurement | Result |
|---|---|
| Decode | 142.9 tok/s |
| Words per second | 95.2 words/s |
| Characters per second | 667.7 chars/s |
| Decode, code prompt | 160.6 tok/s |
| Decode, prose prompt | 125.2 tok/s |
| Spread of the decode runs (CV) | 4.0 % |
| Spread of the code runs (CV) | 2.2 % |
| Spread of the prose runs (CV) | 4.0 % |
| Decode, window 1 | 138.8 tok/s |
| Prefill at 2K | 1650.7 tok/s |
| Prefill at 8K | 3494.4 tok/s |
| Time to first token at 2K | 1.2 s |
| Time to first token at 8K | 2.3 s |
| Draft acceptance rate | 55.0 % |
| Mean accepted length | 2.5 tokens |
| 1 request at once (total) | 104.9 tok/s |
| 2 at once (total) | 104.5 tok/s |
| 4 at once (total) | 106.5 tok/s |
| 8 at once (total) | 106.7 tok/s |
| 1 request at once (per request) | 104.9 tok/s |
| 2 at once (per request) | 78.0 tok/s |
| 4 at once (per request) | 44.3 tok/s |
| 8 at once (per request) | 24.0 tok/s |
| Server slots (--parallel 1 set by us; the start line confirms 1 stream) | 1 slot |
Long prompts
| Prompt length | Prefill | Time to first token | Decode, code prompt | Decode, prose prompt |
|---|---|---|---|---|
| 16K tokens | 3557.3 tok/s | 4.6 s | 156.3 tok/s | 134.9 tok/s |
Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer (empty file, no valid result).
What ran. Own patches on an unmerged pull request head, not an official TensorFold release: the head 06bdb53c (commit 06bdb53c8d225661f2129064a97a44b58b80cbad) of BlivionIaG's pull request #1443, one step of a chain of pull requests (#1441 to #1446) that are open and not merged; the base of the chain is Ash's TensorFold main branch at commit 22fb6014, not the 1.2.7 release. We built the binary ourselves from that head with our own patches on top (sha256 f9f4a57d, recorded with the run; it reports its own version as 1.2.6, the version of the base) with zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc (plus -j8 and our own cache and output folders). The own patches are ours; they are not by Ash or BlivionIaG. The patch files are not published. The model is Qwen3.8-Flash-Next in turboderp's EXL3 pack at 3.05 bits per weight (turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439), with its MTP head for drafts. The experts of the model sit in pinned host memory: 49 layers have offloaded experts, the bank in host memory is 43.21 GiB (copied in 43.4 s); the card holds the rest of the model and a pool of expert lines per layer. This setup is our tuning (the page of our tuning with 1 stream: 253 expert lines from free card memory, the engine's default server reserve) plus own patches and the head's line-delta setting; it is not the setup of the pull request author, and the pages of the author's start settings, of the 131,072-token window and of our tuning are separate pages.
Own patches. The binary has three own patches on top of the pull request head: P1 (table batch), P2 (LRU eviction) and P3 (hit first). Each is off unless an environment variable switches it on, and all three were on for this run. None of them is meant to change what is computed; the equality checks below are what we ran to look for differences. P1, table batch (TF_FLASHNEXT_TABLE_BATCH=1): when an expert line is reloaded into the pool on the card, the head sends 18 small memory-set operations for the table entries of the pool; the patch collects the entries in a copy in host memory and uploads the whole table with one copy per stage, so the table on the card holds the same bytes as before. P2, LRU eviction (TF_FLASHNEXT_EVICT=lru): when the pool needs room, the line that was used longest ago is replaced, instead of the line with the lowest use counter over the life of the server; this changes which experts sit in the pool, not what is computed. P3, hit first (TF_FLASHNEXT_EXL3_HIT_FIRST=1): in a layer where some of the needed experts are missing from the pool, the experts already on the card are computed at once while the missing ones are copied; the missing ones are computed afterwards and the results are combined; the planner, the copies and the eviction are unchanged. LINE_DELTAS (TF_FLASHNEXT_EXL3_LINE_DELTAS) is not a patch of ours but a setting of the pull request head: uneven expert lines per layer (some pools get more lines than the uniform 253, others fewer, as many added as removed; the start line in the raw files lists the values), so the pools still take 21.45 GiB on the card, the same as on the page of our tuning. The binary also holds a diagnostic option that writes a trace of the expert picks; it was not switched on for this run. The patch files are not published.
Where this run differs from the pull request author's setup. (1) Toolchain: our nvcc is CUDA 13.3 (13.3.73), where the author's receipts (#1441) name nvcc 13.4.92, and our driver is 610.43.02, where they name 615.71.09. (2) Build command: zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc; -Dsm=120 is what the author names for the author's build. (3) Sampling: the speed test sends the model card's instruct profile (temperature 0.7, top_p 0.8, top_k 20, min_p 0) with presence_penalty 0, not the 1.5 that the model card names (Qwen3.8-Flash-Next README line 343), because the CUDA backend of this head rejects any presence penalty other than 0 with HTTP 400 (the message reads "presence_penalty and frequency_penalty are not supported by backend 'cuda': send 0 or leave them out"; the first attempt with 1.5 was refused and measured nothing). (4) The MTP drafts are on (the start line reads 'drafts: MTP + copies'), and the CUDA graph switch is at 0, the recipe's default (docs/flash-next-exl3-graphs.md); the author's script starts the first head of the chain with --no-drafts, the head used here serves with MTP drafts. (5) Streams and window: 1 stream and a window of 32,768 tokens, the same as in the author's start script (tools/exl3_serve.sh @5f089bdf, lines 14 and 15). (6) Expert lines and reserves, as on the page of our tuning: the expert lines are not set, so the server chose 253 uniform lines per layer from free card memory, before the line deltas (21.45 GiB on the card, 24.07 GiB free before), where the author's script sets 95 lines (8.16 GiB on the card on the page of the author's start settings). The memory reserve is also different: TENSORFOLD_MEMORY_RESERVE_GIB is not set, so the server kept its default of 1.0 GiB (the start line reads 'of 1.5 GiB left after a 1.0 GiB reserve'), where the author's script sets 0.5 GiB; and the expert reserve that the author's script sets to 256 MB is not set either. The prompt cache is off and the graph is at 0, as in the author's script. (7) What is new on this page compared with the page of our tuning: the own patches and the line-delta setting described above, switched on by four environment variables; the head, the pack, the box, the window, the stream count, the total number of expert lines (253 uniform-equivalent lines, 21.45 GiB on the card) and the reserves are the same; only the distribution of the lines over the layers differs (line deltas). Dropped because of the server window of 32,768 tokens (as in the author's script): the prefill step at 32K, the context steps at 32K, 64K and 128K, and the long-answer step; they have no value (see below).
Window, streams and expert lines. The window is 32,768 tokens and the server runs 1 stream: the start line recorded in the speed result reads '1 stream at once, 1.34 GiB each at a 32768-token window, of 1.5 GiB left after a 1.0 GiB reserve' (model 27.54 GiB on the card), and GET /v1/models reports a context of 32768. The expert pool has 253 lines per layer of 1.76 MiB, counted as uniform-equivalent lines because of the line deltas (21.45 GiB on the card, 24.07 GiB free before); the server loaded in 76.5 s. The start command is in the Setup section below; it sets the window, the number of streams, graph 0 and the prompt cache off, no expert-line setting and no reserve setting, and the four environment variables of the own patches and the line deltas.
Requests at once. The server has 1 stream, so the levels 2, 4 and 8 at once are above its slots: the requests wait in a queue and run one after another, not in parallel. The total stays near the value of a single request (104.9, 104.5, 106.5 and 106.7 tok/s with 1, 2, 4 and 8 requests), and the speed of each request falls as the queue grows (104.9, 78.0, 44.3 and 24.0 tok/s). All requests carry the same prompt.
Dropped steps (reason: the server window of 32,768 tokens, as in the author's script): the prefill step at 32K, the context steps at 32K, 64K and 128K, and the long-answer step. Long prompts: only 16K ran. The tool checked each size before it sent anything: a prompt of 32K, 64K or 128K tokens plus the 512 reply tokens is larger than the 32,768-token window that the server reports, so no request was sent for those sizes and they have no value (the register rows speed2_ctx_d32768_fits, speed2_ctx_d65536_fits and speed2_ctx_d131072_fits are 0). The prefill series has two sizes, 2K and 8K; the size 32K was not run because it does not fit the window. The 16K context step ran with an exact prompt length: prefill 3,557 tok/s, first token after 4.6 s.
Long answer test: no result. The test asks for one reply of up to 32,768 tokens; the window is 32,768 tokens and the prompt takes part of it, so the request cannot be answered: the stream ended at once with no output (the output file l1-output.txt is empty). The step is marked invalid, has no value and is not in the register.
The headline 142.9 tok/s is the mean of the code prompt (160.6) and the prose prompt (125.2), medians of two runs each (spread (CV) of 2.2 % and 4.0 %). The code prompt has 1,260 and 1,261 tokens and the prose prompt 1,351 and 1,353 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 55.0 % (1,228 of 2,234 drafted tokens, mean accepted length 2.50 tokens; counted from the engine's counters over the four decode runs).
Compared with our tuning (our own register job on the page of our tuning with 1 stream: window 32,768 tokens, 1 stream, 253 expert lines, same head, same pack, same box and the same settings, without the own patches and the line deltas): the headline decode value there is 130.5 tok/s, here it is 142.9 tok/s, which is 9.5 % higher (computed from the raw values of the two runs; one run each, and the spread of the decode runs (CV) is 2.2 % and 4.0 % in this run and 2.7 % and 5.8 % in the run of our tuning, code prompt and prose prompt); the code prompt there is 142.4 tok/s and here 160.6, the prose prompt there 118.6 and here 125.2. The prefill series is the same size on both pages (2K and 8K).
Equality checks we ran (probes on separate server starts, not register runs; their raw files are not published here). They use two client tools of the repo: tools/bench_concurrent.py (levels 1, 2 and 4, 128 tokens, 3 repetitions, a code prompt and a chat prompt, each at temperature 0 and at temperature 1, options --alone and --serial) and tools/bench_openai.py (a code prompt and a chat prompt, 128 tokens, 3 repetitions, temperature 0 and 1). (1) On a separate start with the same binary and the same four environment variables as this run, each reply was identical to the same request run alone (option --alone, compared by the hash of the token ids), and the alone runs were identical to runs with drafts switched off (option --serial): 114 equal, 0 unequal, 0 failed. With 1 stream the requests wait in a queue and run one after another, so this check says little about requests at the same time. (2) On the same start, all 96 reply hashes that the two clients recorded (fixed prompts and seeds; 84 from tools/bench_concurrent.py, 12 from tools/bench_openai.py) were equal to the hashes of the same checks on the unmodified pull request head, the head of the page of our tuning. (3) Three further starts of the same binary with only some of the switches on (the hit-first patch together with the LRU eviction, with and without the table batch, and the line deltas with the table batch and the LRU eviction but without the hit-first patch) also gave 114 equal, 0 unequal, 0 failed and 96 of 96 equal hashes each. (4) Long prompts: with the table batch, the LRU eviction and the hit-first patch on and the line deltas off, the replies to prompts of 16,068 and 10,958 tokens, and to three short requests, were the same text as on two starts with the patches switched off and with only the table batch and the LRU eviction on (temperature 0, at most 200 tokens). No long-prompt check was run with the line deltas on. In the equality checks we ran the outputs were identical; this is what these checks show, not more.
Sampling. The speed test sent temperature 0.7, top_p 0.8, top_k 20, min_p 0 and presence_penalty 0 with every request (the instruct profile of the model card, with the one deviation named above), seed 42, cache_prompt false and the flag for no thinking (enable_thinking false). Every speed request carries a nonce, and the cache check of the tool found no cached tokens in any of the 45 requests that it checked.
Foreign load. The load guard of the speed test (any other process above 25 % of a core for two samples in a row) threw away and repeated no step: 0 of 23 steps. Not published: the guard's sample file, the quiet-check tables and the single requests of the speed run.
Exact command we measured withcommand details↓
You do not need this to run the model. It is the exact command of our measurement, so the setup and the command are documented.
python3 speed2.py --base-url http://127.0.0.1:35307/v1 --label fn5090-ourpatch2-1112-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/fn5090-ourpatch2-1112-speed-r0/fn5090-ourpatch2-1112-speed-r0/speed2 --contract $RUN/fn5090-ourpatch2-1112-speed-r0/fn5090-ourpatch2-1112-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/flashnext-5090-1112/pack-3.05bpw_h5_ng5/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0} --sampling-source Qwen/Qwen3.8-Flash-Next README.md:343 (Instruct/non-thinking) and :342 (Thinking), the same lines in the pack turboderp/Qwen3.8-Flash-Next-exl3 README.md:342-343 and :586-587; generation_config.json (1.0/20/0.95) is the thinking default and is not used for the instruct profile. Seed 42 for speed and quality. DEVIATION for instruct: presence_penalty 0 instead of 1.5 of the model card, because the CUDA backend of this pull request head rejects it (HTTP 400 'presence_penalty and frequency_penalty are not supported by backend cuda: send 0 or leave them out', attempt Nov 12, 2026 15:15, nothing measured). --model qwen3.8-flash-next-exl3-3.05bpw --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source the author's script tools/exl3_serve.sh:15 @5f089bdf: '--parallel "${PARALLEL:-1}"' (default 1); server start line '1 stream at once, 1.34 GiB each at a 32768-token window' --server-ctx 32768 --server-ctx-source the author's script tools/exl3_serve.sh:14 @5f089bdf: '--context "${CONTEXT:-32768}"' (window 32768 as in the author's script); server start line '... context: 32768' (probe Nov 12, 2026).Quality
Not run: no Quality v2 test (MMLU-Pro, GPQA, IFEval, LiveCodeBench, long text) was run on this setup, so the page shows no quality number.
Also made this possible
- Model
Qwen (Qwen3.8-Flash-Next)@Alibaba_Qwen Hugging Face GitHub Website - EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)
turboderp@turboderp_ Hugging Face GitHub - Engine (TensorFold; the pull request builds on its main branch)
Ash Hart@ashxhart GitHub Hugging Face Website
Exact command
What you need and how to start it, step by step. The results below come from exactly this setup.
What you need
- Hardware
- 1x NVIDIA GeForce RTX 5090 32 GB (one GPU); the model's experts sit in pinned host memory. Host: AMD Ryzen 9 9950X, 96 GB RAM installed, PCIe 5.0 x16. Driver 610.43.02, CUDA 13.3 (nvcc 13.3 for the build), kernel 6.12.95+deb13-amd64
- Model
- Qwen/Qwen3.8-Flash-Next, EXL3 3.05 bpw (turboderp/Qwen3.8-Flash-Next-exl3 @69e33439, branch 3.05bpw_h5_ng5), MTP head
- EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)
- huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3
- Engine
- Own patches on an unmerged pull request head, not an official TensorFold release (head 06bdb53c of BlivionIaG's pull request chain on Ash's TensorFold, built by us with our own patches, which are not published); turboderp's EXL3 3.05 bpw pack with the experts in host memory; our tuning plus own patches; window 32,768 tokens, 1 stream — the program that loads the model and answers requests · github.com/ashhart/TensorFold
Steps
Get the engine and the model files from the list above. The stock pull request head without the own patches is not this setup, and the patch files are not published.
Start the server with this command. It loads the model into memory.
env CUDA_VISIBLE_DEVICES=0 \ TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> \ TF_FLASHNEXT_EXL3_GRAPH=0 \ TF_FLASHNEXT_TABLE_BATCH=1 \ TF_FLASHNEXT_EVICT=lru \ TF_FLASHNEXT_EXL3_HIT_FIRST=1 \ TF_FLASHNEXT_EXL3_LINE_DELTAS=0:+178,1:+163,2:+128,4:+27,5:+56,6:+98,7:+90,8:+7,9:-26,10:+48,11:-17,12:+36,13:+50,14:-1,15:-129,16:-117,17:-17,18:+68,19:-15,20:-17,21:+43,22:+74,23:-39,24:+4,25:-37,26:+12,27:-27,28:+42,29:+20,30:-25,31:-138,32:-113,33:-23,34:+36,35:-30,36:-40,37:-5,38:+45,39:-101,40:-92,41:-58,42:-10,43:-56,44:-31,45:-37,46:-21,47:-3 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> \ --backend cuda \ --name qwen3.8-flash-next-exl3-3.05bpw \ --port <port> \ --context 32768 \ --parallel 1 \ --no-update-check \ --prompt-cache-gib 0What the parts mean
CUDA_VISIBLE_DEVICES=0which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints--parallel 1how many requests it answers at the same time--name qwen3.8-flash-next-exl3-3.05bpwthe model ID the server advertises to clients
Start text for this setupcommand details↓
You do not need this to run the model. It is the start text of this setup, based on the setup card of this run, with local paths, machine names and addresses shortened or replaced by placeholders, notes added on some pages, and the engine build in front, so the setup and the command are documented.
TensorFold native (Zig), own patches on an unmerged pull request head, not an official TensorFold release: head 06bdb53c of BlivionIaG's pull request #1443 (base main 22fb6014), with three own patches (P1 table batch, P2 LRU eviction, P3 hit first) and the setting LINE_DELTAS of the pull request head (uneven expert lines per layer) switched on, -Dsm=120, nvcc 13.3; MTP drafts on; engine author Ash (ashhart); the own patches are ours, not by Ash or BlivionIaG; the patch files are not published; env CUDA_VISIBLE_DEVICES=0 TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> TF_FLASHNEXT_EXL3_GRAPH=0 TF_FLASHNEXT_TABLE_BATCH=1 TF_FLASHNEXT_EVICT=lru TF_FLASHNEXT_EXL3_HIT_FIRST=1 TF_FLASHNEXT_EXL3_LINE_DELTAS=0:+178,1:+163,2:+128,4:+27,5:+56,6:+98,7:+90,8:+7,9:-26,10:+48,11:-17,12:+36,13:+50,14:-1,15:-129,16:-117,17:-17,18:+68,19:-15,20:-17,21:+43,22:+74,23:-39,24:+4,25:-37,26:+12,27:-27,28:+42,29:+20,30:-25,31:-138,32:-113,33:-23,34:+36,35:-30,36:-40,37:-5,38:+45,39:-101,40:-92,41:-58,42:-10,43:-56,44:-31,45:-37,46:-21,47:-3 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> --backend cuda --name qwen3.8-flash-next-exl3-3.05bpw --port <port> --context 32768 --parallel 1 --no-update-check --prompt-cache-gib 0 (binary built from the pull request head 06bdb53c with our own patches on top, which are not published, with: zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc; our tuning: no expert-line setting, no expert reserve and no memory reserve set, so the server chose the expert lines from free memory; the four environment variables switch on the own patches and the line deltas)Context and sampling: 32,768 tokens, 1 stream (--parallel 1), expert lines chosen from free memory (the server chose 253 lines per layer before the line deltas; the author's script sets 95), memory reserve left at the default 1.0 GiB (the author's script sets 0.5 GiB), graph 0 (the recipe default), MTP drafts on, prompt cache off (--prompt-cache-gib 0), seed 42, every request with a nonce; own patches P1, P2 and P3 (table batch, LRU eviction, hit first) and the head's setting LINE_DELTAS (uneven expert lines per layer) switched on. Sampling sent with every request: the instruct profile of the model card (temperature 0.7, top_p 0.8, top_k 20, min_p 0) with presence_penalty 0 instead of 1.5, because the CUDA backend of this head rejects 1.5 (HTTP 400); thinking off in every request (enable_thinking false). Own patches on an unmerged pull request head, not an official TensorFold release; the patch files are not published. Hardware: 1x NVIDIA GeForce RTX 5090 32 GB (one GPU); the model's experts sit in pinned host memory. Host: AMD Ryzen 9 9950X, 96 GB RAM installed, PCIe 5.0 x16. Driver 610.43.02, CUDA 13.3 (nvcc 13.3 for the build), kernel 6.12.95+deb13-amd64. All raw files of this setup: raw/v2/qwen38-flash-next-5090-exl3-3-05bpw-tensorfold-pr-06bdb53c-our-patch-blivion.