Qwen3.8-Flash-Next on one RTX 5090: TensorFold from a pull request head, our tuning (more expert lines, 4 streams)

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer (empty file, no valid result).

What ran. Built from an unmerged pull request head, not an official TensorFold release: the head 06bdb53c (commit 06bdb53c8d225661f2129064a97a44b58b80cbad) of BlivionIaG's pull request #1443, one step of a chain of pull requests (#1441 to #1446) that are open and not merged; the base of the chain is Ash's TensorFold main branch at commit 22fb6014, not the 1.2.7 release. We built the binary ourselves from that head (sha256 42a56c11, recorded with the run; it reports its own version as 1.2.6, the version of the base) with zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc (plus -j8 and our own cache and output folders). The model is Qwen3.8-Flash-Next in turboderp's EXL3 pack at 3.05 bits per weight (turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439), with its MTP head for drafts. The experts of the model sit in pinned host memory: 49 layers have offloaded experts, the bank in host memory is 43.21 GiB (copied in 43.5 s); the card holds the rest of the model and a pool of expert lines per layer. This setup is our tuning: more expert lines in free card memory (205 instead of 95), and the engine's default server reserve (1.0 GiB instead of the 0.5 GiB set in the author's start script); outputs identical in the equality check. It is not the setup of the pull request author, and the pages of the author's start settings and of the 131,072-token window are separate pages.

Where this run differs from the pull request author's setup. (1) Toolchain: our nvcc is CUDA 13.3 (13.3.73), where the author's receipts (#1441) name nvcc 13.4.92, and our driver is 610.43.02, where they name 615.71.09. (2) Build command: zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc; -Dsm=120 is what the author names for the author's build. (3) Sampling: the speed test sends the model card's instruct profile (temperature 0.7, top_p 0.8, top_k 20, min_p 0) with presence_penalty 0, not the 1.5 that the model card names (Qwen3.8-Flash-Next README line 343), because the CUDA backend of this head rejects any presence penalty other than 0 with HTTP 400 (the message reads "presence_penalty and frequency_penalty are not supported by backend 'cuda': send 0 or leave them out"; the first attempt with 1.5 was refused and measured nothing). (4) The MTP drafts are on (the start line reads 'drafts: MTP + copies'), and the CUDA graph switch is at 0, the recipe's default (docs/flash-next-exl3-graphs.md); the author's script starts the first head of the chain with --no-drafts, the head used here serves with MTP drafts. (5) Streams and window: 4 streams (--parallel 4 set by us; the start line reads '4 streams at once, 1.34 GiB each at a 32768-token window'), where the author's script runs 1 stream; the window is 32,768 tokens, the same as in the author's start script (tools/exl3_serve.sh @5f089bdf, line 14). (6) Expert lines and reserves, the change that makes this setup 'our tuning': the expert lines are not set, so the server chose 205 lines per layer from free card memory (17.41 GiB on the card, 24.07 GiB free before), where the author's script sets 95 lines (8.16 GiB on the card on the page of the author's start settings); at 1 stream the server chose 253 lines. The memory reserve is also different: TENSORFOLD_MEMORY_RESERVE_GIB is not set, so the server kept its default of 1.0 GiB (the start line reads 'of 5.5 GiB left after a 1.0 GiB reserve'), where the author's script sets 0.5 GiB; and the expert reserve that the author's script sets to 256 MB is not set either. The prompt cache is off and the graph is at 0, as in the author's script. Dropped because of the server window of 32,768 tokens (as in the author's script): the prefill step at 32K, the context steps at 32K, 64K and 128K, and the long-answer step; they have no value (see below).

Window, streams and expert lines. The window is 32,768 tokens and the server runs 4 streams: the start line recorded in the speed result reads '4 streams at once, 1.34 GiB each at a 32768-token window, of 5.5 GiB left after a 1.0 GiB reserve' (model 23.51 GiB on the card), and GET /v1/models reports a context of 32768. The expert pool has 205 lines per layer of 1.76 MiB (17.41 GiB on the card, 24.07 GiB free before); the server loaded in 76.6 s. The start command is in the Setup section below; it sets the window, the number of streams, graph 0 and the prompt cache off, and no expert setting and no reserve setting.

Requests at once. The server has 4 streams, so the levels 1, 2 and 4 at once run in parallel and the level 8 queues: 4 requests run, 4 wait. The total goes from 94.0 tok/s with 1 request to 121.8, 149.4 and 151.3 tok/s with 2, 4 and 8 requests, while the speed of one request falls from 94.0 to 62.6, 38.4 and 25.0 tok/s. All requests carry the same prompt.

Dropped steps (reason: the server window of 32,768 tokens, as in the author's script): the prefill step at 32K, the context steps at 32K, 64K and 128K, and the long-answer step. Long prompts: only 16K ran. The tool checked each size before it sent anything: a prompt of 32K, 64K or 128K tokens plus the 512 reply tokens is larger than the 32,768-token window that the server reports, so no request was sent for those sizes and they have no value (the register rows speed2_ctx_d32768_fits, speed2_ctx_d65536_fits and speed2_ctx_d131072_fits are 0). The prefill series has two sizes, 2K and 8K; the size 32K was not run because it does not fit the window. The 16K context step ran with an exact prompt length: prefill 3,548 tok/s, first token after 4.6 s.

Long answer test: no result. The test asks for one reply of up to 32,768 tokens; the window is 32,768 tokens and the prompt takes part of it, so the request cannot be answered: the stream ended at once with no output (the output file l1-output.txt is empty). The step is marked invalid, has no value and is not in the register.

The headline 119.8 tok/s is the mean of the code prompt (124.6) and the prose prompt (114.9), medians of two runs each (spread (CV) of 1.4 % and 2.9 %). The code prompt has 1,258 and 1,260 tokens and the prose prompt 1,352 and 1,352 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 55.6 % (1,208 of 2,172 drafted tokens, mean accepted length 2.44 tokens; counted from the engine's counters over the four decode runs).

Equality check (client modes alone and serial) on a separate start with the same build and settings: 114 equal, 0 unequal, 0 failed. It is a probe, not a register run, and its raw files are not published here. The probe used its own server start (another port) and the prompts of the repo client tools/bench_concurrent.py (levels 1, 2 and 4, 128 tokens, 3 repetitions, options --alone and --serial; a code prompt and a chat prompt, each at temperature 0 and at temperature 1). Its server log shows the same settings as this run: 205 expert lines per layer, 4 streams, a window of 32,768 tokens and the default server reserve of 1.0 GiB. With 4 streams the requests run at the same time, which is what this check is for. In this probe the outputs were identical between the client modes alone and serial, on a server with 205 expert lines per layer.

Sampling. The speed test sent temperature 0.7, top_p 0.8, top_k 20, min_p 0 and presence_penalty 0 with every request (the instruct profile of the model card, with the one deviation named above), seed 42, cache_prompt false and the flag for no thinking (enable_thinking false). Every speed request carries a nonce, and the cache check of the tool found no cached tokens in any of the 45 requests that it checked.

Foreign load. The load guard of the speed test (any other process above 25 % of a core for two samples in a row) threw away and repeated no step: 0 of 23 steps. Not published: the guard's sample file, the quiet-check tables and the single requests of the speed run.

python3 speed2.py --base-url http://127.0.0.1:36835/v1 --label fn5090-ourtuning4-1112-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/fn5090-ourtuning4-1112-speed-r0/fn5090-ourtuning4-1112-speed-r0/speed2 --contract $RUN/fn5090-ourtuning4-1112-speed-r0/fn5090-ourtuning4-1112-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/flashnext-5090-1112/pack-3.05bpw_h5_ng5/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0} --sampling-source Qwen/Qwen3.8-Flash-Next README.md:343 (Instruct/non-thinking) and :342 (Thinking), the same lines in the pack turboderp/Qwen3.8-Flash-Next-exl3 README.md:342-343 and :586-587; generation_config.json (1.0/20/0.95) is the thinking default and is not used for the instruct profile. Seed 42 for speed and quality. DEVIATION for instruct: presence_penalty 0 instead of 1.5 of the model card, because the CUDA backend of this pull request head rejects it (HTTP 400 'presence_penalty and frequency_penalty are not supported by backend cuda: send 0 or leave them out', attempt Nov 12, 2026 15:15, nothing measured). --model qwen3.8-flash-next-exl3-3.05bpw --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 4 --server-slots-source OUR TUNING (4 streams): --parallel 4 set by us (the author's script tools/exl3_serve.sh:15 uses the default 1); server start line '4 streams at once, 1.34 GiB each at a 32768-token window' --server-ctx 32768 --server-ctx-source the author's script tools/exl3_serve.sh:14 @5f089bdf: '--context "${CONTEXT:-32768}"' (window 32768 as in the author's script); server start line '... context: 32768' (probe Nov 12, 2026).

Quality

Not run: no Quality v2 test (MMLU-Pro, GPQA, IFEval, LiveCodeBench, long text) was run on this setup, so the page shows no quality number.

Also made this possible

• ModelQwen (Qwen3.8-Flash-Next)@Alibaba_Qwen Hugging Face GitHub Website
• EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)turboderp@turboderp_ Hugging Face GitHub
• Engine (TensorFold; the pull request builds on its main branch)Ash Hart@ashxhart GitHub Hugging Face Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one GPU); the model's experts sit in pinned host memory. Host: AMD Ryzen 9 9950X, 96 GB RAM installed, PCIe 5.0 x16. Driver 610.43.02, CUDA 13.3 (nvcc 13.3 for the build), kernel 6.12.95+deb13-amd64

Model

Qwen/Qwen3.8-Flash-Next, EXL3 3.05 bpw (turboderp/Qwen3.8-Flash-Next-exl3 @69e33439, branch 3.05bpw_h5_ng5), MTP head

EXL3 quant (pack 3.05bpw_h5_ng5, revision 69e33439)

huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3

Engine

Built from an unmerged pull request head, not an official TensorFold release (head 06bdb53c of BlivionIaG's pull request chain on Ash's TensorFold); turboderp's EXL3 3.05 bpw pack with the experts in host memory; our tuning: more expert lines in free card memory, outputs identical in the equality check; window 32,768 tokens, 4 streams — the program that loads the model and answers requests · github.com/ashhart/TensorFold

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory.

   Copy

   env CUDA_VISIBLE_DEVICES=0 \
     TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> \
     TF_FLASHNEXT_EXL3_GRAPH=0 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> \
     --backend cuda \
     --name qwen3.8-flash-next-exl3-3.05bpw \
     --port <port> \
     --context 32768 \
     --parallel 4 \
     --no-update-check \
     --prompt-cache-gib 0

   What the parts mean

   • CUDA_VISIBLE_DEVICES=0which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints
   • --parallel 4how many requests it answers at the same time
   • --name qwen3.8-flash-next-exl3-3.05bpwthe model ID the server advertises to clients

TensorFold native (Zig), built from an unmerged pull request head, not an official TensorFold release: head 06bdb53c of BlivionIaG's pull request #1443 (base main 22fb6014), -Dsm=120, nvcc 13.3; MTP drafts on; engine author Ash (ashhart); env CUDA_VISIBLE_DEVICES=0 TENSORFOLD_CHECKPOINT_SHA256=<sha256 of model.safetensors.index.json and config.json of the pack> TF_FLASHNEXT_EXL3_GRAPH=0 tensorfold-native serve <folder of turboderp/Qwen3.8-Flash-Next-exl3, branch 3.05bpw_h5_ng5, revision 69e33439> --backend cuda --name qwen3.8-flash-next-exl3-3.05bpw --port <port> --context 32768 --parallel 4 --no-update-check --prompt-cache-gib 0 (binary built from the pull request head 06bdb53c with: zig build native -Dsm=120 -Dnvcc=<CUDA 13.3 folder>/bin/nvcc; our tuning: no expert-line setting, no expert reserve and no memory reserve set, so the server chose the expert lines from free memory)