Qwen3.8-Flash-Next on one RTX 5090: TensorFold from a pull request head with own patches (our patch 3, four patches, 1 stream)

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer (empty file, no valid result).

What ran. Own patches on an unmerged pull request head, not an official TensorFold release: the head 06bdb53c (commit 06bdb53c8d225661f2129064a97a44b58b80cbad) of BlivionIaG's pull request #1443, one step of a chain of pull requests (#1441 to #1446) that are open and not merged; the base of the chain is Ash's TensorFold main branch at commit 22fb6014, not the 1.2.7 release. We built the binary ourselves from that head with our own patches on top (sha256 90c51b49, recorded with the run; it reports its own version as 1.2.6, the version of the base) with zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc (plus -j8 and our own cache and output folders). The own patches are ours; they are not by Ash or BlivionIaG. The patch files are not published. The model is Qwen3.8-Flash-Next in turboderp's EXL3 pack at 3.05 bits per weight (turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439), with its MTP head for drafts. The experts of the model sit in pinned host memory: 49 layers have offloaded experts, the bank in host memory is 43.21 GiB (copied in 44.6 s); the card holds the rest of the model and a pool of expert lines per layer. This setup is our tuning (the page of our tuning with 1 stream: 253 expert lines from free card memory, the engine's default server reserve) plus four own patches and the head's line-delta setting; it is the setup of the page of our patch (three own patches) with one more own patch, P4. It is not the setup of the pull request author, and the pages of the author's start settings, of the 131,072-token window, of our tuning and of our patch are separate pages.

Own patches. The binary has four own patches on top of the pull request head: P1 (table batch), P2 (LRU eviction), P3 (hit first) and P4 (prefetch of n-gram rows). The first three are the patches of the page of our patch. Each is off unless an environment variable switches it on, and all four were on for this run. None of them is meant to change what is computed; the equality checks below are what we ran to look for differences. P1, table batch (TF_FLASHNEXT_TABLE_BATCH=1): when an expert line is reloaded into the pool on the card, the head sends 18 small memory-set operations for the table entries of the pool; the patch collects the entries in a copy in host memory and uploads the whole table with one copy per stage, so the table on the card holds the same bytes as before. P2, LRU eviction (TF_FLASHNEXT_EVICT=lru): when the pool needs room, the line that was used longest ago is replaced, instead of the line with the lowest use counter over the life of the server; this changes which experts sit in the pool, not what is computed. P3, hit first (TF_FLASHNEXT_EXL3_HIT_FIRST=1): in a layer where some of the needed experts are missing from the pool, the experts already on the card are computed at once while the missing ones are copied; the missing ones are computed afterwards and the results are combined; the planner, the copies and the eviction are unchanged. P4, prefetch of n-gram rows (TF_FLASHNEXT_PLE_PREFETCH=1): the verify step gathers rows of the n-gram embedding table of the pack for the tokens of its window; the table is read from the model file with memory mapping, so a row that is not yet in the operating system's page cache is read from the drive at that moment. With P4 on, the draft chain asks the kernel (madvise with MADV_WILLNEED) to read the pages of the coming rows into the page cache earlier, while the graphics card computes the next draft step; it only asks, and the verify step then finds the same bytes sooner, so what is computed is not meant to change. The binary also holds a built-in timer for the host work of a round (TF_FLASHNEXT_GAPTRACE), which is off by default and was off for this run. LINE_DELTAS (TF_FLASHNEXT_EXL3_LINE_DELTAS) is not a patch of ours but a setting of the pull request head: uneven expert lines per layer (some pools get more lines than the uniform 253, others fewer, as many added as removed; the start line in the raw files lists the values), so the pools still take 21.45 GiB on the card, the same as on the page of our tuning. The line-delta values were fitted on routing traces that include the speed2 prompts (one trace for the fit and a second trace with the same prompt set for the check), so the speed test of this page uses the same prompts that the values were fitted on. The binary also holds a diagnostic option that writes a trace of the expert picks; it was not switched on for this run. The patch files are not published.

Where this run differs from the pull request author's setup. (1) Toolchain: our nvcc is CUDA 13.3 (13.3.73), where the author's receipts (#1441) name nvcc 13.4.92, and our driver is 610.43.02, where they name 615.71.09. (2) Build command: zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc; -Dsm=120 is what the author names for the author's build. (3) Sampling: the speed test sends the model card's instruct profile (temperature 0.7, top_p 0.8, top_k 20, min_p 0) with presence_penalty 0, not the 1.5 that the model card names (Qwen3.8-Flash-Next README line 343), because the CUDA backend of this head rejects any presence penalty other than 0 with HTTP 400 (the message reads "presence_penalty and frequency_penalty are not supported by backend 'cuda': send 0 or leave them out"; the first attempt with 1.5 was refused and measured nothing). (4) The MTP drafts are on (the start line reads 'drafts: MTP + copies'), and the CUDA graph switch is at 0, the recipe's default (docs/flash-next-exl3-graphs.md); the author's script starts the first head of the chain with --no-drafts, the head used here serves with MTP drafts. (5) Streams and window: 1 stream and a window of 32,768 tokens, the same as in the author's start script (tools/exl3_serve.sh @5f089bdf, lines 14 and 15). (6) Expert lines and reserves, as on the page of our tuning: the expert lines are not set, so the server chose 253 uniform lines per layer from free card memory, before the line deltas (21.45 GiB on the card, 24.07 GiB free before), where the author's script sets 95 lines (8.16 GiB on the card on the page of the author's start settings). The memory reserve is also different: TENSORFOLD_MEMORY_RESERVE_GIB is not set, so the server kept its default of 1.0 GiB (the start line reads 'of 1.5 GiB left after a 1.0 GiB reserve'), where the author's script sets 0.5 GiB; and the expert reserve that the author's script sets to 256 MB is not set either. The prompt cache is off and the graph is at 0, as in the author's script. (7) What is new on this page compared with the page of our tuning: the own patches and the line-delta setting described above, switched on by five environment variables; the head, the pack, the box, the window, the stream count, the total number of expert lines (253 uniform-equivalent lines, 21.45 GiB on the card) and the reserves are the same; only the distribution of the lines over the layers differs (line deltas). Compared with the page of our patch, the only change is the patch P4 and its switch (the other three patches and the line deltas are the same). Dropped because of the server window of 32,768 tokens (as in the author's script): the prefill step at 32K, the context steps at 32K, 64K and 128K, and the long-answer step; they have no value (see below).

Window, streams and expert lines. The window is 32,768 tokens and the server runs 1 stream: the start line recorded in the speed result reads '1 stream at once, 1.34 GiB each at a 32768-token window, of 1.5 GiB left after a 1.0 GiB reserve' (model 27.54 GiB on the card), and GET /v1/models reports a context of 32768. The expert pool has 253 lines per layer of 1.76 MiB, counted as uniform-equivalent lines because of the line deltas (21.45 GiB on the card, 24.07 GiB free before); the server loaded in 78.0 s. The start command is in the Setup section below; it sets the window, the number of streams, graph 0 and the prompt cache off, no expert-line setting and no reserve setting, and the five environment variables of the own patches and the line deltas.

Requests at once. The server has 1 stream, so the levels 2, 4 and 8 at once are above its slots: the requests wait in a queue and run one after another, not in parallel. The total stays near the value of a single request (112.7, 111.4, 110.2 and 110.4 tok/s with 1, 2, 4 and 8 requests), and the speed of each request falls as the queue grows (112.7, 83.9, 45.8 and 24.8 tok/s). All requests carry the same prompt.

Dropped steps (reason: the server window of 32,768 tokens, as in the author's script): the prefill step at 32K, the context steps at 32K, 64K and 128K, and the long-answer step. Long prompts: only 16K ran. The tool checked each size before it sent anything: a prompt of 32K, 64K or 128K tokens plus the 512 reply tokens is larger than the 32,768-token window that the server reports, so no request was sent for those sizes and they have no value (the register rows speed2_ctx_d32768_fits, speed2_ctx_d65536_fits and speed2_ctx_d131072_fits are 0). The prefill series has two sizes, 2K and 8K; the size 32K was not run because it does not fit the window. The 16K context step ran with an exact prompt length: prefill 3,566 tok/s, first token after 4.6 s.

Long answer test: no result. The test asks for one reply of up to 32,768 tokens; the window is 32,768 tokens and the prompt takes part of it, so the request cannot be answered: the stream ended at once with no output (the output file l1-output.txt is empty). The step is marked invalid, has no value and is not in the register.

The headline 157.2 tok/s is the mean of the code prompt (161.9) and the prose prompt (152.6), medians of two runs each (spread (CV) of 2.6 % and 1.8 %). The code prompt has 1,258 and 1,260 tokens and the prose prompt 1,355 and 1,355 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 55.3 % (1,262 of 2,282 drafted tokens, mean accepted length 2.61 tokens; counted from the engine's counters over the four decode runs).

Compared with our patch (our own register job on the page of our patch: same head, pack, box, window, stream count and expert lines, with the three own patches P1 to P3 and the line deltas, without P4): the headline decode value there is 142.9 tok/s, here it is 157.2 tok/s, which is 10.0 % higher (computed from the raw values of the two runs, one run each); the code prompt there is 160.6 tok/s and here 161.9 (0.8 %), the prose prompt there 125.2 and here 152.6 (21.9 %). The gain is almost all in the prose prompt; the spread of the decode runs (CV) is 2.6 % and 1.8 % in this run and 2.2 % and 4.0 % in the run of our patch, code prompt and prose prompt. The prefill values at 2K and 8K differ from the run of our patch by 0.9 % and 0.4 % and the 16K context step by 0.2 %; these small differences are not stated as an effect of P4. The requests-at-once values are not compared here: with 1 stream they are one request at a time.

Compared with our tuning (our own register job on the page of our tuning with 1 stream: window 32,768 tokens, 1 stream, 253 expert lines, same head, same pack, same box and the same settings, without the own patches and the line deltas): the headline decode value there is 130.5 tok/s, here it is 157.2 tok/s, which is 20.4 % higher (computed from the raw values of the two runs; one run each, and the spread of the decode runs (CV) is 2.7 % and 5.8 % in the run of our tuning, code prompt and prose prompt); the code prompt there is 142.4 tok/s and here 161.9, the prose prompt there 118.6 and here 152.6. The prefill series is the same size on both pages (2K and 8K).

Equality checks we ran (probes on separate server starts, not register runs; their raw files are not published here). They use two client tools of the repo: tools/bench_concurrent.py (levels 1, 2 and 4, 128 tokens, 3 repetitions, a code prompt and a chat prompt, each at temperature 0 and at temperature 1, options --alone and --serial) and tools/bench_openai.py (a code prompt and a chat prompt, 128 tokens, 3 repetitions, temperature 0 and 1). (1) Three starts of the same binary as this run, each with the three earlier own patches and the line deltas on: with the new switch of P4 off, with it on, and with it on and 4 streams (--parallel 4). In each start, each reply was identical to the same request run alone (option --alone, compared by the hash of the token ids), and the alone runs were identical to runs with drafts switched off (option --serial): 114 equal, 0 unequal, 0 failed in every start. With 1 stream the requests wait in a queue and run one after another, so the 1-stream starts say little about requests at the same time; the start with 4 streams does. (2) On each of these three starts, all 96 reply hashes that the two clients recorded (fixed prompts and seeds; 84 from tools/bench_concurrent.py, 12 from tools/bench_openai.py) were equal to the hashes of the same checks on the unmodified pull request head, the head of the page of our tuning; the 125 comparable token hashes in each server log (the other 18 requests of the single-request client are not comparable) were equal as well. (3) Pick trace at temperature 0 (6 prompts, 300 tokens each, with the table batch, the LRU eviction and the hit-first patch on, line deltas off): the 6 replies and the 28,456 recorded pick lines were identical, by checksum, on the binary of the page of our patch, on this binary with the P4 switch off and on this binary with it on. (4) Long prompts: with all patches and the P4 switch on, the replies to prompts of 16,068 and 10,958 tokens, and to three short requests, were the same text as on an earlier start with the own patches switched off (temperature 0, at most 200 tokens). The host tests of the engine (zig test) were not run on this build. In the equality checks we ran the outputs were identical; this is what these checks show, not more.

Sampling. The speed test sent temperature 0.7, top_p 0.8, top_k 20, min_p 0 and presence_penalty 0 with every request (the instruct profile of the model card, with the one deviation named above), seed 42, cache_prompt false and the flag for no thinking (enable_thinking false). Every speed request carries a nonce, and the cache check of the tool found no cached tokens in any of the 45 requests that it checked.

Foreign load. The load guard of the speed test (any other process above 25 % of a core for two samples in a row) threw away and repeated no step: 0 of 23 steps. Not published: the guard's sample file, the quiet-check tables and the single requests of the speed run.

python3 speed2.py --base-url http://127.0.0.1:43611/v1 --label fn5090-ourpatch3-1112-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/fn5090-ourpatch3-1112-speed-r0/fn5090-ourpatch3-1112-speed-r0/speed2 --contract $RUN/fn5090-ourpatch3-1112-speed-r0/fn5090-ourpatch3-1112-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/flashnext-5090-1112/pack-3.05bpw_h5_ng5/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0} --sampling-source Qwen/Qwen3.8-Flash-Next README.md:343 (Instruct/non-thinking) and :342 (Thinking), the same lines in the pack turboderp/Qwen3.8-Flash-Next-exl3 README.md:342-343 and :586-587; generation_config.json (1.0/20/0.95) is the thinking default and is not used for the instruct profile. Seed 42 for speed and quality. DEVIATION for instruct: presence_penalty 0 instead of 1.5 of the model card, because the CUDA backend of this pull request head rejects it (HTTP 400 'presence_penalty and frequency_penalty are not supported by backend cuda: send 0 or leave them out', attempt Nov 12, 2026 15:15, nothing measured). --model qwen3.8-flash-next-exl3-3.05bpw --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source the author's script tools/exl3_serve.sh:15 @5f089bdf: '--parallel "${PARALLEL:-1}"' (default 1); server start line '1 stream at once, 1.34 GiB each at a 32768-token window' --server-ctx 32768 --server-ctx-source the author's script tools/exl3_serve.sh:14 @5f089bdf: '--context "${CONTEXT:-32768}"' (window 32768 as in the author's script); server start line '... context: 32768' (probe Nov 12, 2026).

Quality

Not run: no Quality v2 test (MMLU-Pro, GPQA, IFEval, LiveCodeBench, long text) was run on this setup, so the page shows no quality number.

Also made this possible

• ModelQwen (Qwen3.8-Flash-Next)@Alibaba_Qwen Hugging Face GitHub Website
• EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)turboderp@turboderp_ Hugging Face GitHub
• Engine (TensorFold; the pull request builds on its main branch)Ash Hart@ashxhart GitHub Hugging Face Website

Exact command

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one GPU); the model's experts sit in pinned host memory. Host: AMD Ryzen 9 9950X, 96 GB RAM installed, PCIe 5.0 x16. Driver 610.43.02, CUDA 13.3 (nvcc 13.3 for the build), kernel 6.12.95+deb13-amd64

Model

Qwen/Qwen3.8-Flash-Next, EXL3 3.05 bpw (turboderp/Qwen3.8-Flash-Next-exl3 @69e33439, branch 3.05bpw_h5_ng5), MTP head

EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)

huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3

Engine

Own patches on an unmerged pull request head, not an official TensorFold release (head 06bdb53c of BlivionIaG's pull request chain on Ash's TensorFold, built by us with our own patches, which are not published); turboderp's EXL3 3.05 bpw pack with the experts in host memory; our tuning plus four own patches; window 32,768 tokens, 1 stream — the program that loads the model and answers requests · github.com/ashhart/TensorFold

Steps

1. Get the engine and the model files from the list above. The stock pull request head without the own patches is not this setup, and the patch files are not published.

2. Start the server with this command. It loads the model into memory.

   Copy

   env CUDA_VISIBLE_DEVICES=0 \
     TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> \
     TF_FLASHNEXT_EXL3_GRAPH=0 \
     TF_FLASHNEXT_TABLE_BATCH=1 \
     TF_FLASHNEXT_EVICT=lru \
     TF_FLASHNEXT_EXL3_HIT_FIRST=1 \
     TF_FLASHNEXT_PLE_PREFETCH=1 \
     TF_FLASHNEXT_EXL3_LINE_DELTAS=0:+178,1:+163,2:+128,4:+27,5:+56,6:+98,7:+90,8:+7,9:-26,10:+48,11:-17,12:+36,13:+50,14:-1,15:-129,16:-117,17:-17,18:+68,19:-15,20:-17,21:+43,22:+74,23:-39,24:+4,25:-37,26:+12,27:-27,28:+42,29:+20,30:-25,31:-138,32:-113,33:-23,34:+36,35:-30,36:-40,37:-5,38:+45,39:-101,40:-92,41:-58,42:-10,43:-56,44:-31,45:-37,46:-21,47:-3 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> \
     --backend cuda \
     --name qwen3.8-flash-next-exl3-3.05bpw \
     --port <port> \
     --context 32768 \
     --parallel 1 \
     --no-update-check \
     --prompt-cache-gib 0

   What the parts mean

   • CUDA_VISIBLE_DEVICES=0which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints
   • --parallel 1how many requests it answers at the same time
   • --name qwen3.8-flash-next-exl3-3.05bpwthe model ID the server advertises to clients

TensorFold native (Zig), own patches on an unmerged pull request head, not an official TensorFold release: head 06bdb53c of BlivionIaG's pull request #1443 (base main 22fb6014), with four own patches (P1 table batch, P2 LRU eviction, P3 hit first, P4 prefetch of n-gram rows) and the setting LINE_DELTAS of the pull request head (uneven expert lines per layer) switched on, -Dsm=120, nvcc 13.3; MTP drafts on; engine author Ash (ashhart); the own patches are ours, not by Ash or BlivionIaG; the patch files are not published; env CUDA_VISIBLE_DEVICES=0 TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> TF_FLASHNEXT_EXL3_GRAPH=0 TF_FLASHNEXT_TABLE_BATCH=1 TF_FLASHNEXT_EVICT=lru TF_FLASHNEXT_EXL3_HIT_FIRST=1 TF_FLASHNEXT_PLE_PREFETCH=1 TF_FLASHNEXT_EXL3_LINE_DELTAS=0:+178,1:+163,2:+128,4:+27,5:+56,6:+98,7:+90,8:+7,9:-26,10:+48,11:-17,12:+36,13:+50,14:-1,15:-129,16:-117,17:-17,18:+68,19:-15,20:-17,21:+43,22:+74,23:-39,24:+4,25:-37,26:+12,27:-27,28:+42,29:+20,30:-25,31:-138,32:-113,33:-23,34:+36,35:-30,36:-40,37:-5,38:+45,39:-101,40:-92,41:-58,42:-10,43:-56,44:-31,45:-37,46:-21,47:-3 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> --backend cuda --name qwen3.8-flash-next-exl3-3.05bpw --port <port> --context 32768 --parallel 1 --no-update-check --prompt-cache-gib 0 (binary built from the pull request head 06bdb53c with our own patches on top, which are not published, with: zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc; our tuning: no expert-line setting, no expert reserve and no memory reserve set, so the server chose the expert lines from free memory; the five environment variables switch on the four own patches and the line deltas)