Qwen3.8-Flash-Next on one RTX 5090: TensorFold from a pull request head, window 131,072

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer.

What ran. Built from an unmerged pull request head, not an official TensorFold release: the head 06bdb53c (commit 06bdb53c8d225661f2129064a97a44b58b80cbad) of BlivionIaG's pull request #1443, one step of a chain of pull requests (#1441 to #1446) that are open and not merged; the base of the chain is Ash's TensorFold main branch at commit 22fb6014, not the 1.2.7 release. We built the binary ourselves from that head (sha256 42a56c11, recorded with the run; it reports its own version as 1.2.6, the version of the base) with zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc (plus -j8 and our own cache and output folders). The model is Qwen3.8-Flash-Next in turboderp's EXL3 pack at 3.05 bits per weight (turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439), with its MTP head for drafts. The experts of the model sit in pinned host memory: 49 layers have offloaded experts, the bank in host memory is 43.21 GiB (copied in 45.1 s); the card holds the rest of the model (12.26 GiB) and a pool of expert lines per layer.

Where this run differs from the pull request author's setup. (1) Toolchain: our nvcc is CUDA 13.3 (13.3.73), where the author's receipts (#1441) name nvcc 13.4.92, and our driver is 610.43.02, where they name 615.71.09. (2) Build command: zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc; -Dsm=120 is what the author names for the author's build. (3) Sampling: the speed test sends the model card's instruct profile (temperature 0.7, top_p 0.8, top_k 20, min_p 0) with presence_penalty 0, not the 1.5 that the model card names (Qwen3.8-Flash-Next README line 343), because the CUDA backend of this head rejects any presence penalty other than 0 with HTTP 400 (the message reads "presence_penalty and frequency_penalty are not supported by backend 'cuda': send 0 or leave them out"; the first attempt with 1.5 was refused and measured nothing). (4) The MTP drafts are on (the start line reads 'drafts: MTP + copies'), and the CUDA graph switch is at 0, the recipe's default (docs/flash-next-exl3-graphs.md). (5) Window and streams: 131,072 tokens and 4 streams, more than the 32,768 tokens and 1 stream of the author's own start script (tools/exl3_serve.sh); the card has room for them and nothing was lowered. (6) Expert lines: not set, so the server chose 71 lines per layer from free memory (6.14 GiB on the card), where the author's script sets 95. The same setup in the author's own start settings has its own page.

Window, streams and expert lines. The window is 131,072 tokens and the server runs 4 streams: the start line recorded in the speed result reads '4 streams at once, 4.16 GiB each at a 131072-token window, of 16.8 GiB left after a 1.0 GiB reserve' (model 12.26 GiB on the card), and GET /v1/models reports a context of 131072. The expert pool has 71 lines per layer of 1.76 MiB (6.14 GiB on the card, 24.04 GiB free before); the server loaded in 78.5 s. The start command is in the Setup section below; it sets the window, the number of streams, graph 0 and the prompt cache off, and no expert setting.

Requests at once. The server has 4 streams, so the levels 1, 2 and 4 at once run in parallel and the level 8 queues: 4 requests run, 4 wait. The total goes from 62.0 tok/s with 1 request to 72.6, 78.1 and 76.1 tok/s with 2, 4 and 8 requests, while the speed of one request falls from 62.0 to 36.7, 19.7 and 13.0 tok/s. All requests carry the same prompt.

Long prompts. The prefill series ran at 2K, 8K and 32K tokens in three runs each, and the context steps at 16K, 32K and 64K tokens; every prompt length was counted exactly with the server's tokenizer. The 128K step ran at 130,048 tokens, not 131,072: a prompt of 131,072 tokens plus the 512 reply tokens does not fit the 131,072-token window, so the step ran at the window minus 1,024 tokens (130,048), which the rules of the test allow down to 126,976 tokens (the register row speed2_ctx_d131072_depth_tokens holds the real depth). Prefill at 130,048 tokens: 3,446 tok/s, first token after 37.7 s; decode there: 72.8 tok/s (code prompt) and 73.1 tok/s (prose prompt).

Long answer. The test asks for one long reply (up to 32,768 tokens) to a short prompt: 8,492 tokens in 101.6 s, ended by the model itself (finish reason stop). The file l1-output.txt holds the reply.

The headline 71.9 tok/s is the mean of the code prompt (77.1) and the prose prompt (66.7), medians of two runs each (spread (CV) of 3.4 % and 0.5 %). The code prompt has 1,257 and 1,260 tokens and the prose prompt 1,355 and 1,355 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 56.0 % (1,243 of 2,220 drafted tokens, mean accepted length 2.55 tokens; counted from the engine's counters over the four decode runs).

Sampling. The speed test sent temperature 0.7, top_p 0.8, top_k 20, min_p 0 and presence_penalty 0 with every request (the instruct profile of the model card, with the one deviation named above), seed 42, cache_prompt false and the flag for no thinking (enable_thinking false). Every speed request carries a nonce, and the cache check of the tool found no cached tokens in any of the 61 requests that it checked.

Foreign load. The load guard of the speed test (any other process above 25 % of a core for two samples in a row) threw away and repeated no step: 0 of 38 steps. Not published: the guard's sample file, the quiet-check tables and the single requests of the speed run.

python3 speed2.py --base-url http://127.0.0.1:38493/v1 --label fn5090-1112-speed-r2 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/fn5090-1112-speed-r2/fn5090-1112-speed-r2/speed2 --contract $RUN/fn5090-1112-speed-r2/fn5090-1112-speed-r2/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/flashnext-5090-1112/pack-3.05bpw_h5_ng5/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0} --sampling-source Qwen/Qwen3.8-Flash-Next README.md:343 (Instruct/non-thinking) and :342 (Thinking), the same lines in the pack turboderp/Qwen3.8-Flash-Next-exl3 README.md:342-343 and :586-587; generation_config.json (1.0/20/0.95) is the thinking default and is not used for the instruct profile. Seed 42 for speed and quality. DEVIATION for instruct: presence_penalty 0 instead of 1.5 of the model card, because the CUDA backend of this pull request head rejects it (HTTP 400 'presence_penalty and frequency_penalty are not supported by backend cuda: send 0 or leave them out', attempt Nov 12, 2026 15:15, nothing measured). --model qwen3.8-flash-next-exl3-3.05bpw --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 4 --server-slots-source our setting --parallel 4; the author's script tools/exl3_serve.sh @5f089bdf: '--parallel "${PARALLEL:-1}"' (default 1, settable by environment variable), the author's receipts (#1441) ran with four overlapping requests; server start line at a context of 131072: '4 streams at once, 4.16 GiB each' --server-ctx 131072 --server-ctx-source start argument --context 131072; server start line (probe Nov 12): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 12.26 GiB; 4 streams at once, 4.16 GiB each at a 131072-token window, of 16.8 GiB left after a 1.0 GiB reserve ... context: 131072'. The author's script tools/exl3_serve.sh (first head of the chain) sets --context ${CONTEXT:-32768}: 131072 is a raise by us (the model's window is 262144), not a cut.

Quality

Not run: no Quality v2 test (MMLU-Pro, GPQA, IFEval, LiveCodeBench, long text) was run on this setup, so the page shows no quality number.

Also made this possible

• ModelQwen (Qwen3.8-Flash-Next)@Alibaba_Qwen Hugging Face GitHub Website
• EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)turboderp@turboderp_ Hugging Face GitHub
• Engine (TensorFold; the pull request builds on its main branch)Ash Hart@ashxhart GitHub Hugging Face Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one GPU); the model's experts sit in pinned host memory. Host: AMD Ryzen 9 9950X, 96 GB RAM installed, PCIe 5.0 x16. Driver 610.43.02, CUDA 13.3 (nvcc 13.3 for the build), kernel 6.12.95+deb13-amd64

Model

Qwen/Qwen3.8-Flash-Next, EXL3 3.05 bpw (turboderp/Qwen3.8-Flash-Next-exl3 @69e33439, branch 3.05bpw_h5_ng5), MTP head

EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)

huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3

Engine

Built from an unmerged pull request head, not an official TensorFold release (head 06bdb53c of BlivionIaG's pull request chain on Ash's TensorFold); turboderp's EXL3 3.05 bpw pack with the experts in host memory; window 131,072 tokens, 4 streams, expert lines from free memory — the program that loads the model and answers requests · github.com/ashhart/TensorFold

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory.

   Copy

   env CUDA_VISIBLE_DEVICES=0 \
     TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> \
     TF_FLASHNEXT_EXL3_GRAPH=0 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> \
     --backend cuda \
     --name qwen3.8-flash-next-exl3-3.05bpw \
     --port <port> \
     --context 131072 \
     --parallel 4 \
     --no-update-check \
     --prompt-cache-gib 0

   What the parts mean

   • CUDA_VISIBLE_DEVICES=0which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints
   • --parallel 4how many requests it answers at the same time
   • --name qwen3.8-flash-next-exl3-3.05bpwthe model ID the server advertises to clients

TensorFold native (Zig), built from an unmerged pull request head, not an official TensorFold release: head 06bdb53c of BlivionIaG's pull request #1443 (base main 22fb6014), -Dsm=120, nvcc 13.3; MTP drafts on; engine author Ash (ashhart); env CUDA_VISIBLE_DEVICES=0 TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> TF_FLASHNEXT_EXL3_GRAPH=0 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> --backend cuda --name qwen3.8-flash-next-exl3-3.05bpw --port <port> --context 131072 --parallel 4 --no-update-check --prompt-cache-gib 0 (binary built from the pull request head 06bdb53c with: zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc; no expert setting: the server chose the expert lines from free memory)