Ash's TensorFold 1.2.6 on one RTX 5090: Qwen3.6-35B-A3B (NVIDIA NVFP4, DFlash2 drafts)

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds; long prompts (later supplementary run of 2026-11-11, same setup): prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests. Measured 2026-11-11. Raw files: speed result · speed contract · log · long answer · speed result · speed contract · log.

Recipe as written, with three added flags, one of which changes the window. The speed test ran with the author's recipe: TensorFold 1.2.6 as the official release archive for x86_64 Linux (tensorfold-1.2.6-linux-x86_64.tar.gz, published 10 Nov 2026, 12:51 UTC, sha256 a8abf9b8, the digest GitHub lists for that file; I downloaded the archive and its binary tensorfold-native has the sha256 e2c14a41 that the measuring machine recorded; no patch, no container) and the command of docs/recipes/qwen3.6-moe.md line 163, tensorfold serve nvidia/Qwen3.6-35B-A3B-NVFP4 --drafter incoai/Qwen3.6-35B-A3B-DFlash2. The recorded command starts the binary tensorfold-native of that archive (README line 79 defines tensorfold as a call of tensorfold-native, line 97 allows a local checkpoint directory) with the local folders of the two checkpoints. I added a display name (--name), the port (--port 8131, a free port) and --context 131072 (see Window and streams). The checkpoints are the revisions the recipe names (1355db6a for NVIDIA's NVFP4 checkpoint, line 141, and 51ef7b69 for the DFlash2 drafter, line 162). I copied the two folders from another of my machines and checked them file by file against the names, sizes and hashes that Hugging Face lists for those revisions (17 and 6 files; the marker line of that check, dated 3 Nov 2026, is in the start text recorded with the run). Neither the recipe nor the README names an RTX 5090 for this model. The README lists this model for one GB10 (line 199: 'One GB10; 20.3 GiB on the device, 20.9 GiB with the drafter'), and the pull request that added the DFlash2 part of the recipe gives its gate for one DGX Spark (GB10, sm_121, CUDA 13.0). The README says that the CUDA release images include sm_80, sm_86, sm_120 and sm_121 (line 170), and the card of this run records that the release archive for x86_64 reports the chip nvidia-sm120 and the family qwen3_5_moe with modelopt-nvfp4 in its capabilities output (not re-run for this page, so not part of the raw files). I draw no conclusion beyond what this run shows. Window and streams. The author's command sets neither a context nor a stream count (recipe line 163), and the README names a default window of 32,768 tokens for Qwen3.8-27B only (line 307), none for this model. I did not start this release without --context, so I cannot say here which window the server would choose on this card with the command exactly as written. My tests need more than 32,768 tokens in one window (LiveCodeBench answers of up to 32,768 tokens, the 32K long-text prompts, the long-prompt steps from 32K), so I added --context 131072; the README says that --context bounds prompt plus reply (line 307); the model's own window is 262,144 tokens (the checkpoint's max_position_embeddings). This is my deviation from the author's command, and every number on this page was measured with it, except that the memory test restarts the server once per size with --context N. With this window the server chose the number of streams itself (--parallel auto asks for up to eight streams as memory allows, README lines 104 and 345): its start line reads '3 streams at once, 2.96 GiB each at a 131072-token window, of 9.4 GiB left after a 1.0 GiB reserve' (the model takes 20.47 GiB), and GET /v1/models reports a context of 131072 (start line and speed result, recorded with the run). Long prompts. The 128K step could not use 131,072 prompt tokens: with 512 reply tokens that is 131,584 tokens against the window of 131,072. The test then takes the window minus 1,024, which is 130,048 tokens, as its rule says (valid from 126,976). The row for 128K in the table of long prompts was measured at 130,048 tokens; the other three steps (16K, 32K, 64K) have the exact length. The memory test below reaches the full window of 131,072 tokens with a prompt of 124,518 tokens. The prefill series of this card has two sizes, 2K and 8K, so the tiles for prefill and first token stop at 8K; the prefill and first-token values for 16K, 32K, 64K and 128K are in the table of long prompts. The long-prompt steps were measured twice. The speed run itself contained them (raw folder tfq36-5090-126-1111-speed-r0), and a supplementary run of the same day repeated only these steps (raw folder tfq36-5090-126-1111-ctx-speed-r0-ctx, same start command). The table shows the supplementary run. The values of the speed run are: 16K: 392.2 tok/s (code) and 359.7 tok/s (prose), prefill 17485.1 tok/s, first token after 0.9 s; 32K: 327.4 tok/s (code) and 343.0 tok/s (prose), prefill 15179.9 tok/s, first token after 2.2 s; 64K: 320.0 tok/s (code) and 328.3 tok/s (prose), prefill 12076.0 tok/s, first token after 5.4 s; 128K step (130,048 tokens): 297.9 tok/s (code) and 308.6 tok/s (prose), prefill 8546.1 tok/s, first token after 15.2 s. Requests at once. The server has 3 streams. The levels 1 and 2 are inside them. '4 at once' sends 4 requests to 3 streams (1 waits) and '8 at once' sends 8 requests to 3 streams (5 wait); the tool logs both as measured anyway. So these two rows measure the waiting requests as well as the streams: '8 at once' does not mean 8 streams, and the rows for 4 and 8 at once are not comparable with the rows for 1 and 2 at once. The total rises from 358.2 tok/s with 1 request to 617.6 tok/s with 8 requests, while the speed of one request falls from 358.2 to 108.0 tok/s. Sampling is the model card's setting for replies without thinking (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5; model card line 665), sent with the request flag for no thinking and seed 42. The chat template thinks by default, and the author's command passes no sampling flag, so the server's own default is temperature 1.0, top_p 0.95, top_k 20 (start line; generation_config.json of the checkpoint). The headline 404.4 tok/s is the mean of the code prompt (417.8) and the prose prompt (391.1), medians of two runs each (spread (CV) of 3.2 % and 6.2 %). The code prompt has 1,260 and 1,262 tokens and the prose prompt 1,352 and 1,354 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 18.8 % (1,492 of 7,940 drafted tokens, mean accepted length 3.7 tokens). The long-answer test asks for one reply of up to 32,768 tokens; the reply had 4,462 tokens and ended with a stop after 7.6 seconds. The recipe's own numbers (lines 166 to 169: 130.3 tok/s at one stream and 131.3, 161.4 and 198.0 tok/s in total with 2, 4 and 8 users) come from the fleet benchmark of the recipe (its script bench.py, a prompt of 4.2K tokens, 512 tokens); pull request #1674 gives the same figures in its gate, taken on one DGX Spark (GB10). My speed test uses another method (two prompts, medians of two runs, 512 tokens) on another machine, so these figures are not a like-for-like reference for this card, and I compare nothing against them. The release notes of v1.2.6 (line 17) say: 'Qwen3.6-35B-A3B MoE prompts on a GB10: 9.9-14.1% less time, from 2K to 64K.' That is a measurement on a GB10 as well, and I compare nothing against it. Foreign load. The load guard (any other process above 25 % of a core for two samples in a row, sampled twice a second) was set to ignore one kernel thread, kcompactd0 (memory compaction). In all 35 steps of the speed run and all 16 steps of the supplementary run it found no other process above its limit, so no step was repeated; the highest other process in any step was the Docker daemon (dockerd) at 16.0 % of a core. The quiet check of the speed run (one sample about every 10 seconds) took 18 samples and the one of the supplementary run 13 samples; all of them show only the kernel's memory thread kswapd0, at most 1.8 % of a core.

python3 speed2.py --base-url http://127.0.0.1:42489/v1 --label tfq36-5090-126-1111-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/tfq36-5090-126-1111-speed-r0/tfq36-5090-126-1111-speed-r0/speed2 --contract $RUN/tfq36-5090-126-1111-speed-r0/tfq36-5090-126-1111-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/models/qwen36-nvfp4/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source Qwen/Qwen3.6-35B-A3B README.md:665 (instruct/non-thinking: temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence_penalty 1.5) and nvidia checkpoint generation_config.json @1355db6a (temperature 1.0, top_p 0.95, top_k 20 = the engine default the author's command uses, README :`Sampling defaults come from the checkpoint's generation_config.json`; presence_penalty 0.0 as in the incoai DFlash2 evaluation setup, the README.md of incoai/Qwen3.6-35B-A3B-DFlash2, section Evaluation: temperature 1.0, top_p 0.95, top_k 20, presence 0.0, thinking enabled). The author's recipe passes no sampling flag. Seed 42 for speed/quality. --model qwen3.6-35b-a3b-nvfp4-dflash2 --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 3 --server-slots-source DEVIATION CONTEXT 131072 (author's command plus --context 131072, reason: a window of 32768 tokens does not hold LCB, RULER 32K and the context depths from 32K); streams = the server's own choice at this window, read from the startup line at the start test Nov 11, 2026 (release 1.2.6): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 20.47 GiB; 3 streams at once, 2.96 GiB each at a 131072-token window, of 9.4 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' (TensorFold v1.2.6 README.md:104 and :345, --parallel auto, the recipe sets none); the recipe command sets no --parallel --server-ctx 131072 --server-ctx-source DEVIATION from the author's command (named openly, reason: a window of 32768 tokens does not hold LCB, RULER 32K and the depths from 32K): tensorfold-native serve ... --context 131072 (README.md:307 (v1.2.6): --context bounds prompt plus reply), start script qw36-5090-ctl-126.sh with Q2_CTX=131072 (the same switch the fit part uses)

# Long prompts: supplementary run of 2026-11-11 (raw folder tfq36-5090-126-1111-ctx-speed-r0-ctx)
python3 speed2.py --base-url http://127.0.0.1:41993/v1 --label tfq36-5090-126-1111-ctx-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/tfq36-5090-126-1111-ctx-speed-r0/tfq36-5090-126-1111-ctx-speed-r0/speed2 --contract $RUN/tfq36-5090-126-1111-ctx-speed-r0/tfq36-5090-126-1111-ctx-speed-r0/speed2/speed-contract.json --parts ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/models/qwen36-nvfp4/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source Qwen/Qwen3.6-35B-A3B README.md:665 (instruct/non-thinking: temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence_penalty 1.5) and nvidia checkpoint generation_config.json @1355db6a (temperature 1.0, top_p 0.95, top_k 20 = the engine default the author's command uses, README :`Sampling defaults come from the checkpoint's generation_config.json`; presence_penalty 0.0 as in the incoai DFlash2 evaluation setup, the README.md of incoai/Qwen3.6-35B-A3B-DFlash2, section Evaluation: temperature 1.0, top_p 0.95, top_k 20, presence 0.0, thinking enabled). The author's recipe passes no sampling flag. Seed 42 for speed/quality. --model qwen3.6-35b-a3b-nvfp4-dflash2 --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 3 --server-slots-source DEVIATION CONTEXT 131072 (author's command plus --context 131072, reason: a window of 32768 tokens does not hold LCB, RULER 32K and the context depths from 32K); streams = the server's own choice at this window, read from the startup line at the start test Nov 11, 2026 (release 1.2.6): '[tensorfold] CUDA sm_120 device 0 (NVIDIA GeForce RTX 5090): model 20.47 GiB; 3 streams at once, 2.96 GiB each at a 131072-token window, of 9.4 GiB left after a 1.0 GiB reserve; prompts in passes of up to 4096 rows, a 8.0 GiB prompt cache in host memory with 0 MiB of device staging; own glue kernels' (TensorFold v1.2.6 README.md:104 and :345, --parallel auto, the recipe sets none); the recipe command sets no --parallel --server-ctx 131072 --server-ctx-source DEVIATION from the author's command (named openly, reason: a window of 32768 tokens does not hold LCB, RULER 32K and the depths from 32K): tensorfold-native serve ... --context 131072 (README.md:307 (v1.2.6): --context bounds prompt plus reply), start script qw36-5090-ctl-126.sh with Q2_CTX=131072 (the same switch the fit part uses) --supplement-of tfq36-5090-126-1111

How well it answers

All seven Quality v2 tests were run with the window of 131,072 tokens and 3 requests at a time, the 3 streams of the server. Thinking was off in six of them; GPQA 30 ran with thinking on.

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-11.

Note from the run: MMLU-Pro and GPQA Diamond ran with an answer limit of 1,024 tokens. My usual limit for these two tests is 48; I raised it so that a reply with a reason after the letter is not counted as cut off. The questions, the seed 42 and the first-letter extraction are the usual ones. Every answer carried a letter the tool could read (median 2 tokens). IFEval ran with 4,096 tokens (prompt-level strict), LiveCodeBench with 32,768, long text with 128 and GPQA 30 with thinking with 16,384. Cut off at their limits: 0 of 400 MMLU-Pro, 0 of 198 GPQA and 0 of 200 IFEval answers, 4 of 40 LiveCodeBench answers (all 4 at 32,768 tokens) and none in the long-text tests; 5 of the 30 thinking answers were cut off at the limit. A cut-off answer counts as wrong. The questions of each test ran 3 at a time, the 3 streams of the server (same questions, limits and seed for every question), so the median time per question of the thinking test was measured under that load: it is not a speed value. GPQA Diamond ran from the public mirror fingertap/GPQA-Diamond (198 questions), because the original is gated and no Hugging Face token was available.

LiveCodeBench: the 40 newest tasks of the set (contest dates 2025-03-15 to 2025-04-06), pass@1, the code runs in a sandbox without network. Of the 18 answers that failed, the 4 cut-off ones count as wrong, and one more answer (a reply of 288 tokens) had no code. By difficulty, 7 of 9 easy, 8 of 11 medium and 7 of 20 hard tasks passed (counted from the answer file, which the site does not publish). Long text at 8K and 32K, 4 tasks of 10 each, scored strictly: multi-key retrieval 10 of 10 and 10 of 10, variable tracking 2 of 10 and 0 of 10, common-word extraction 10 of 10 and 10 of 10, multi-hop question answering 8 of 10 and 4 of 10. A variable-tracking task counts only when all variables are named (mean fraction 0.6 at 8K and 0.25 at 32K), and a multi-hop answer counts when the expected answer, normalised (lower case, no punctuation or articles), appears in the reply as a run of words. The 32K prompts have 32,779 and 32,780 tokens. Three of the 80 prompts are one token away from the exact length (8,191, 8,193 and 32,767 tokens; the tool marks them not exact).

Thinking test (GPQA 30, thinking on): 30 of the GPQA questions, with the thinking profile of the checkpoint (temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence penalty 0; the presence penalty is the one of the evaluation setup on the DFlash2 card, line 72), seed 42, a limit of 16,384 tokens. 5 of the 30 were still thinking at the limit and count as wrong; of the 25 answers that finished, 22 are right. The Qwen model card recommends an output length of 32,768 tokens for most queries and 81,920 tokens for highly complex problems (line 1012), so 16,384 tokens is below that and 73.3 % is the score at this limit. The Qwen model card lists a presence penalty of 1.5 for thinking mode on general tasks (line 663); this test did not use it. The author's command passes no sampling flag. The median of 6,505 tokens and 18.4 seconds per question covers all 30 questions, including the 5 at the limit.

Sampling. The tests with thinking off sent the model card's setting for replies without thinking (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5; model card line 665) with the request flag for no thinking; every request of the quality tests sent cache_prompt false and seed 42. One run of each test, no repetition.

Foreign load. These tests have no load guard; their quiet check lists the busiest other process about every 10 seconds. LiveCodeBench and long text: 70 samples, 3 of them above 25 % of a core (97.8, 94.7 and 97.2 %: the code sandbox of the LiveCodeBench test itself, a Python process running the solution under test, /work/solution.py, and its runner /work/sandbox_runner.py; nothing was repeated). The other 67 samples show only kernel threads (kswapd0 and one kworker), at most 2.9 %. MMLU-Pro, GPQA Diamond and IFEval: 13 samples, all of them kswapd0, at most 1.8 %. GPQA 30 with thinking: 27 samples, all of them kswapd0, at most 1.8 %. The card was empty (2 MiB in use) before the server started. Not published: the LiveCodeBench answer file and its code and test folders (the site's filter holds back source files), the quiet-check tables, and the prompt files of the long-text tests; the summaries, logs and contracts of all tests and the answer files of the other tests are published.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen36-35b-a3b-nvfp4-dflash2-tensorfold-1.2.6-ash-recipe-5090-ctx131072-1111.json --parts mmlupro --job tfq36-5090-126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen36-35b-a3b-nvfp4-dflash2-tensorfold-1.2.6-ash-recipe-5090-ctx131072-1111.json --parts gpqa --job tfq36-5090-126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen36-35b-a3b-nvfp4-dflash2-tensorfold-1.2.6-ash-recipe-5090-ctx131072-1111.json --parts ifeval --job tfq36-5090-126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen36-35b-a3b-nvfp4-dflash2-tensorfold-1.2.6-ash-recipe-5090-ctx131072-1111.json --parts lcb --job tfq36-5090-126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen36-35b-a3b-nvfp4-dflash2-tensorfold-1.2.6-ash-recipe-5090-ctx131072-1111.json --parts ruler --job tfq36-5090-126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen36-35b-a3b-nvfp4-dflash2-tensorfold-1.2.6-ash-recipe-5090-ctx131072-1111.json --parts ruler --job tfq36-5090-126-1111

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen36-35b-a3b-nvfp4-dflash2-tensorfold-1.2.6-ash-recipe-5090-ctx131072-1111.json --parts think --job tfq36-5090-126-1111

How much context fits

Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed at the largest size was not measured. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. Measured 2026-11-11. Raw files: result of every step · log · contract.

Note from the run: The largest size tested is 131,072 tokens (128K), the top step of the series and the window of the speed and quality tests. It loaded and answered, so it is not necessarily the most this card could hold. The model's own window is 262,144 tokens; I did not test sizes above 131,072 tokens. How this differs from the speed and quality tests: the memory test starts the server once per size with the author's command plus --context N (a copy of the start script takes N from an environment variable). At 131,072 tokens it is the same command as the server of the speed and quality tests, whose start line reads 3 streams at once. The six server logs of this test are not published: they hold only lines of our start script, not the server's start text, so the number of streams at the smaller windows is not shown. Peak memory is the memory in use on the whole card (nvidia-smi; it was empty before each start, 2 MiB) and includes the weights (20.47 GiB at the window of 131,072 tokens). It rises from 25.04 GiB at 4K to 29.41 GiB at 32K and stays between 29.37 and 29.75 GiB from 32K to 128K; the peak at 64K is 0.04 GiB lower than the one at 32K, and I did not log the streams at the smaller windows, so I cannot say why. Decode speed at these sizes is not measured (the server returns no timings).

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen36-35b-a3b-nvfp4-dflash2-tensorfold-1.2.6-ash-recipe-5090-ctx131072-1111.json --parts fit --job tfq36-5090-126-1111

Also made this possible

• Code for this model in TensorFold, all merged by Ash: NVFP4 serving (pull request #1662), DFlash2 drafts (#1674) and the CUDA router change for prompts of 257 or more rows that this release lists (#2537)LlaudneyGitHub
• The recipe names this Python port (pull request #345) as the source of its rules: bf16 activations for the W4A16 layers, the shared expert last in the table, and the MTP layer drafting through the target's headGgilbyGitHub
• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• Quant (NVFP4, NVIDIA Model Optimizer, revision 1355db6a)NVIDIAHugging Face GitHub Website
• Draft model (DFlash2, revision 51ef7b69)Inco AI@inco_ai Hugging Face GitHub Website
• Speculative decoding method (DFlash: the paper that the drafter's model card asks to cite; its GitHub link there is z-lab/dflash)z-lab@zhijianliu_ GitHub Hugging Face

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one GPU, the whole model in video memory). Host: AMD Ryzen 9 9950X, 92 GB RAM. Driver 610.43.02, kernel 6.12.95+deb13-amd64

Model

nvidia/Qwen3.6-35B-A3B-NVFP4 @1355db6a (ModelOpt NVFP4, 35B-A3B MoE) + incoai/Qwen3.6-35B-A3B-DFlash2 @51ef7b69 (drafter, affine-4 at load)

Quant (NVFP4, NVIDIA Model Optimizer, revision 1355db6a)

huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4

Draft model (DFlash2, revision 51ef7b69)

huggingface.co/incoai/Qwen3.6-35B-A3B-DFlash2

Engine

1x NVIDIA GeForce RTX 5090 32 GB, one GPU (the whole model in video memory). Host: AMD Ryzen 9 9950X, 92 GB RAM. Driver 610.43.02, kernel 6.12.95+deb13-amd64. Official release for x86_64 (native CUDA, sm_120), NVIDIA's NVFP4 checkpoint with DFlash2 drafts, a window of 131,072 tokens (--context 131072 added to the author's command), 3 streams — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.6

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8131.

   Copy

   tensorfold-native serve qwen36-nvfp4 \
     --drafter qwen36-dflash2 \
     --name qwen3.6-35b-a3b-nvfp4-dflash2 \
     --port 8131 \
     --context 131072

   The file names are the ones we used. Run the command in the folder that holds them, or put the full path in front of each name.

   What the parts mean

   • --port 8131the network port the server listens on
   • --drafter qwen36-dflash2the draft model: it guesses the next tokens and the main model checks them; without it the model decodes plain
   • --name qwen3.6-35b-a3b-nvfp4-dflash2the model ID the server advertises to clients
   • --context 131072how many tokens a prompt and its reply may use together; the author's command does not set it, I added it, and the reason is in the start text below

TensorFold native (Zig) 1.2.6 official release archive tensorfold-1.2.6-linux-x86_64.tar.gz (sha256 a8abf9b8), qwen3_5_moe CUDA family (modelopt-nvfp4, chip nvidia-sm120 per capabilities --json), DFlash2 drafter, window 131072 tokens (--context 131072 added to the author's command), streams chosen by the server, 1x RTX 5090 (sm_120); tensorfold-native serve $HOME/models/qwen36-nvfp4 --drafter $HOME/models/qwen36-dflash2 --name qwen3.6-35b-a3b-nvfp4-dflash2 --port 8131 --context 131072 (recipe: tensorfold serve nvidia/Qwen3.6-35B-A3B-NVFP4 --drafter incoai/Qwen3.6-35B-A3B-DFlash2; I added --name, --port and --context 131072; wrapper $HOME/qwen36-5090-0311/qw36-5090-ctl-126.sh)