Qwen3.8-Flash-Next on one RTX 5090: TensorFold from a pull request head, the author's start settings

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer (empty file, no valid result).

What ran. Built from an unmerged pull request head, not an official TensorFold release: the head 06bdb53c (commit 06bdb53c8d225661f2129064a97a44b58b80cbad) of BlivionIaG's pull request #1443, one step of a chain of pull requests (#1441 to #1446) that are open and not merged; the base of the chain is Ash's TensorFold main branch at commit 22fb6014, not the 1.2.7 release. We built the binary ourselves from that head (sha256 42a56c11, recorded with the run; it reports its own version as 1.2.6, the version of the base) with zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc (plus -j8 and our own cache and output folders). The model is Qwen3.8-Flash-Next in turboderp's EXL3 pack at 3.05 bits per weight (turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439), with its MTP head for drafts. The experts of the model sit in pinned host memory: 49 layers have offloaded experts, the bank in host memory is 43.21 GiB (copied in 44.2 s); the card holds the rest of the model (14.25 GiB) and a pool of expert lines per layer.

Where this run differs from the pull request author's setup. (1) Toolchain: our nvcc is CUDA 13.3 (13.3.73), where the author's receipts (#1441) name nvcc 13.4.92, and our driver is 610.43.02, where they name 615.71.09. (2) Build command: zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc; -Dsm=120 is what the author names for the author's build. (3) Sampling: the speed test sends the model card's instruct profile (temperature 0.7, top_p 0.8, top_k 20, min_p 0) with presence_penalty 0, not the 1.5 that the model card names (Qwen3.8-Flash-Next README line 343), because the CUDA backend of this head rejects any presence penalty other than 0 with HTTP 400 (the message reads "presence_penalty and frequency_penalty are not supported by backend 'cuda': send 0 or leave them out"; the first attempt with 1.5 was refused and measured nothing). (4) The MTP drafts are on (the start line reads 'drafts: MTP + copies'), and the CUDA graph switch is at 0, the recipe's default (docs/flash-next-exl3-graphs.md). (5) Window and streams: this page uses the author's own start settings from tools/exl3_serve.sh @5f089bdf (lines 12 to 15): window 32,768 tokens, 1 stream, 95 expert lines per layer with a reserve of 256 MB and a memory reserve of 0.5 GiB, prompt cache 0. Dropped because of the server window of 32,768 tokens (the author's script): the prefill step at 32K, the context steps at 32K, 64K and 128K, and the long-answer step; they have no value (see below). The script itself starts the first head of the chain with --no-drafts; the head used here serves with MTP drafts, so the drafts are on. The same head with a window of 131,072 tokens and 4 streams has its own page.

Window, streams and expert lines. The window is 32,768 tokens and the server runs 1 stream: the start line recorded in the speed result reads '1 stream at once, 1.34 GiB each at a 32768-token window, of 15.3 GiB left after a 0.5 GiB reserve' (model 14.25 GiB on the card), and GET /v1/models reports a context of 32768. The expert pool has 95 lines per layer of 1.76 MiB (8.16 GiB on the card, 24.07 GiB free before); the server loaded in 77.5 s. The start command is in the Setup section below; it sets the window, the number of streams, graph 0, the prompt cache off, and the author's three expert settings (95 lines, reserve 256 MB, memory reserve 0.5 GiB).

Requests at once. The server has 1 stream, so the levels 2, 4 and 8 at once are above its slots: the requests wait in a queue and run one after another, not in parallel. The total stays near the value of a single request (69.1, 70.0, 70.7 and 70.7 tok/s with 1, 2, 4 and 8 requests), and the speed of each request falls as the queue grows (69.1, 52.5, 29.4 and 15.9 tok/s). All requests carry the same prompt.

Dropped steps (reason: the server window of 32,768 tokens, the author's script): the prefill step at 32K, the context steps at 32K, 64K and 128K, and the long-answer step. Long prompts: only 16K ran. The tool checked each size before it sent anything: a prompt of 32K, 64K or 128K tokens plus the 512 reply tokens is larger than the 32,768-token window that the server reports, so no request was sent for those sizes and they have no value (the register rows speed2_ctx_d32768_fits, speed2_ctx_d65536_fits and speed2_ctx_d131072_fits are 0). The prefill series has two sizes, 2K and 8K; the size 32K was not run because it does not fit the window. The 16K context step ran with an exact prompt length: prefill 3,525 tok/s, first token after 4.6 s.

Long answer test: no result. The test asks for one reply of up to 32,768 tokens; the window is 32,768 tokens and the prompt takes part of it, so the request cannot be answered: the stream ended at once with no output (the output file l1-output.txt is empty). The step is marked invalid, has no value and is not in the register.

The headline 81.3 tok/s is the mean of the code prompt (85.4) and the prose prompt (77.2), medians of two runs each (spread (CV) of 0.1 % and 2.3 %). The code prompt has 1,260 and 1,261 tokens and the prose prompt 1,349 and 1,352 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 55.0 % (1,254 of 2,278 drafted tokens, mean accepted length 2.59 tokens; counted from the engine's counters over the four decode runs).

Sampling. The speed test sent temperature 0.7, top_p 0.8, top_k 20, min_p 0 and presence_penalty 0 with every request (the instruct profile of the model card, with the one deviation named above), seed 42, cache_prompt false and the flag for no thinking (enable_thinking false). Every speed request carries a nonce, and the cache check of the tool found no cached tokens in any of the 45 requests that it checked.

Foreign load. The load guard of the speed test (any other process above 25 % of a core for two samples in a row) threw away and repeated no step: 0 of 23 steps. Not published: the guard's sample file, the quiet-check tables and the single requests of the speed run.

python3 speed2.py --base-url http://127.0.0.1:38389/v1 --label fn5090-author-1112-speed-r1 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/fn5090-author-1112-speed-r1/fn5090-author-1112-speed-r1/speed2 --contract $RUN/fn5090-author-1112-speed-r1/fn5090-author-1112-speed-r1/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/flashnext-5090-1112/pack-3.05bpw_h5_ng5/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0} --sampling-source Qwen/Qwen3.8-Flash-Next README.md:343 (Instruct/non-thinking) and :342 (Thinking), the same lines in the pack turboderp/Qwen3.8-Flash-Next-exl3 README.md:342-343 and :586-587; generation_config.json (1.0/20/0.95) is the thinking default and is not used for the instruct profile. Seed 42 for speed and quality. DEVIATION for instruct: presence_penalty 0 instead of 1.5 of the model card, because the CUDA backend of this pull request head rejects it (HTTP 400 'presence_penalty and frequency_penalty are not supported by backend cuda: send 0 or leave them out', attempt Nov 12, 2026 15:15, nothing measured). --model qwen3.8-flash-next-exl3-3.05bpw --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source the author's script tools/exl3_serve.sh:15 @5f089bdf: '--parallel "${PARALLEL:-1}"' (default 1); server start line '1 stream at once, 1.34 GiB each at a 32768-token window' --server-ctx 32768 --server-ctx-source the author's script tools/exl3_serve.sh:14 @5f089bdf: '--context "${CONTEXT:-32768}"' (window 32768 as in the author's script); server start line '... context: 32768' (probe Nov 12, 2026).

Quality

Not run: no Quality v2 test (MMLU-Pro, GPQA, IFEval, LiveCodeBench, long text) was run on this setup, so the page shows no quality number.

Also made this possible

• ModelQwen (Qwen3.8-Flash-Next)@Alibaba_Qwen Hugging Face GitHub Website
• EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)turboderp@turboderp_ Hugging Face GitHub
• Engine (TensorFold; the pull request builds on its main branch)Ash Hart@ashxhart GitHub Hugging Face Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one GPU); the model's experts sit in pinned host memory. Host: AMD Ryzen 9 9950X, 96 GB RAM installed, PCIe 5.0 x16. Driver 610.43.02, CUDA 13.3 (nvcc 13.3 for the build), kernel 6.12.95+deb13-amd64

Model

Qwen/Qwen3.8-Flash-Next, EXL3 3.05 bpw (turboderp/Qwen3.8-Flash-Next-exl3 @69e33439, branch 3.05bpw_h5_ng5), MTP head

EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)

huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3

Engine

Built from an unmerged pull request head, not an official TensorFold release (head 06bdb53c of BlivionIaG's pull request chain on Ash's TensorFold); turboderp's EXL3 3.05 bpw pack with the experts in host memory; the pull request author's start settings: window 32,768 tokens, 1 stream, 95 expert lines — the program that loads the model and answers requests · github.com/ashhart/TensorFold

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory.

   Copy

   env CUDA_VISIBLE_DEVICES=0 \
     TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> \
     TF_FLASHNEXT_EXL3_GRAPH=0 \
     TF_FLASHNEXT_EXPERT_LINES=95 \
     TF_FLASHNEXT_EXPERT_RESERVE_MB=256 \
     TENSORFOLD_MEMORY_RESERVE_GIB=0.5 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> \
     --backend cuda \
     --name qwen3.8-flash-next-exl3-3.05bpw \
     --port <port> \
     --context 32768 \
     --parallel 1 \
     --no-update-check \
     --prompt-cache-gib 0

   What the parts mean

   • CUDA_VISIBLE_DEVICES=0which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints
   • --parallel 1how many requests it answers at the same time
   • --name qwen3.8-flash-next-exl3-3.05bpwthe model ID the server advertises to clients

TensorFold native (Zig), built from an unmerged pull request head, not an official TensorFold release: head 06bdb53c of BlivionIaG's pull request #1443 (base main 22fb6014), -Dsm=120, nvcc 13.3; MTP drafts on; engine author Ash (ashhart); env CUDA_VISIBLE_DEVICES=0 TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> TF_FLASHNEXT_EXL3_GRAPH=0 TF_FLASHNEXT_EXPERT_LINES=95 TF_FLASHNEXT_EXPERT_RESERVE_MB=256 TENSORFOLD_MEMORY_RESERVE_GIB=0.5 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> --backend cuda --name qwen3.8-flash-next-exl3-3.05bpw --port <port> --context 32768 --parallel 1 --no-update-check --prompt-cache-gib 0 (binary built from the pull request head 06bdb53c with: zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc; the expert settings are those of the author's start script tools/exl3_serve.sh)