Ash's TensorFold 1.2.5 on two DGX Sparks: Qwen3.8-Flash-Next (NVIDIA NVFP4 checkpoint)

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-10. Raw files: speed result · speed contract · log · long answer.

Requests at once: the server has 8 slots (--parallel 8, which I added to the command of the recipe page as a documented option; see point (3) below). The speed test ran with Ash's official release of TensorFold v1.2.5 (published 8 Nov 2026, 17:02 UTC): the prebuilt Linux/aarch64 archive tensorfold-1.2.5-linux-aarch64.tar.gz from the release page (sha256 1a0192b28d6ab6ec2749f3ff25515379dba4895171e781b30f810afc971ae4f8, the same as the release's own checksum file), not a build of mine; the program inside is the same file on both boxes (sha256 4f7ed2ab...). The model is NVIDIA's own NVFP4 checkpoint nvidia/Qwen3.8-Flash-Next-NVFP4, revision fc694b54 (11 weight files; the sha256 of every file equals its Hugging Face blob name, on both boxes). One box held the files before the run; I copied them to the other over the CX7 link and checked all 11 hashes again. NCCL is version 2.30.7, NVIDIA's own build, as the documentation requires NCCL 2.30 or later. The command is the one of Ash's recipe page ("One or two ranks": serve the checkpoint with --tp 2, rank 1 first, then rank 0 with --name and --host), with the settings listed next. Differences from the author's text, all named here: (1) Presence penalty: the Qwen model card recommends a presence penalty of 1.5 for non-thinking mode, but the official server 1.2.5 refuses presence and frequency penalties with an error (HTTP 400, "send 0 or leave them out"; the v1.2.4 release notes say that unsupported penalties are refused instead of dropped, lines 25 and 134). A first attempt of this test failed on exactly that (nothing of it is on this page). So every request of the speed, instruct and LiveCodeBench parts sends a presence penalty of 0 instead of 1.5 (it is in the sampling line of the exact command below). The thinking test uses 0 as the model card does. This is a forced change from the model card, and it can change the quality numbers of the instruct tests. (2) Window and prompt cache: the documentation of the two-box path (flash-next-cuda.md, lines 20, 31 to 33 and 41) shows the command with --context 65536 and --prompt-cache-gib 0 and says the cache option is required with drafts at --tp 2. The v1.2.4 release notes (line 100) say that at --tp 2 with drafts the prompt states are now kept and the server no longer needs --prompt-cache-gib 0 to start. I followed the release notes and ran the engine's defaults: the whole window of 262,144 tokens and a prompt cache of 16.0 GiB. The documentation page of v1.2.5 still carries the older text. The speed test sends every request with the cache switched off, and none of the 61 checked requests reported cached tokens. (3) --parallel 8 on both ranks: not in the example command, but a documented option (the same number on both ranks; "--parallel auto" selects one request, qwen3.8-flash-next.md, lines 403 to 406); I set it so that 2, 4 and 8 requests at once can be measured. The start line reports 8 streams of 4.69 GiB each at the full window. (4) Status: the documentation names this two-box path experimental, pending independent full-model numerical qualification (flash-next-cuda.md, lines 10 to 20). Read every number on this page as a number from an experimental path. The documentation also says that at --tp 2 each stream's window verifies alone, so concurrent streams do not yet share a verify pass (line 42). The headline 38.8 tok/s is the mean of the code prompt (40.1) and the prose prompt (37.5), medians of two runs each (spread 1.2 % and 9.7 %). Drafts were accepted at a rate of 57.3 % (1,289 of 2,250), 2.71 tokens per step on average. Requests at once: the server has 8 slots, so the four levels (1, 2, 4 and 8 at once) are all inside its slot count. The total rises from 32.4 to 89.0 tok/s for 1 to 8 requests at once; the speed of one request falls from 32.4 to 11.7 tok/s (the 1-request figure includes the prompt time, so it is below the decode speed). The 128K step is a prompt of exactly 131,072 tokens and fits the window; every context step ran. Ash's release notes give their own figures for this model on a GB10 (for example v1.2.5, lines 44 to 46: cold prompts of 4K to 34K tokens take 3.7 to 4.4 % less time); I did not repeat the author's method, so they are not compared with the numbers here. The load guard (a sample every half second) repeated 0 of its 38 steps; the busiest other process in any step was dockerd at 14 % of one core (it repeats a step only when one process stays above 25 % for two samples in a row). The ten-second quiet check had another process above 25 % of one core in 9 of 143 samples (up to 100 %); all 9 were ssh calls of my own watchdog script, which reads the free memory and the rank state of the second box every few seconds and stops both ranks if memory runs low (it never triggered). Both boxes also run their own services.

python3 speed2.py --base-url http://127.0.0.1:35503/v1 --label fn125c-1010-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/fn125c-1010-speed-r0/fn125c-1010-speed-r0/speed2 --contract $RUN/fn125c-1010-speed-r0/fn125c-1010-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/.cache/huggingface/hub/models--nvidia--Qwen3.8-Flash-Next-NVFP4/snapshots/fc694b54fb0174e0913e6adf86691ef85a4ead47/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0} --sampling-source Qwen/Qwen3.8-Flash-Next README.md:343 (instruct, non-thinking: temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence penalty 1.5) and :342 (thinking: temperature 1.0, top_p 0.95, top_k 20); generation_config.json (1.0/20/0.95) is the thinking default only and is not used for instruct. FORCED CHANGE FROM THE MODEL CARD (10 Nov 2026, seen in the first attempt of this test): the official TensorFold 1.2.5 server refuses presence_penalty and frequency_penalty with HTTP 400 (message: presence_penalty and frequency_penalty are not supported by backend cuda: send 0 or leave them out; release notes v1.2.4: Unsupported penalties are refused instead of dropped), so the instruct parts send presence_penalty 0 instead of 1.5. The thinking parts are unchanged (presence penalty 0) --model Qwen3.8-Flash-Next --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 8 --server-slots-source --parallel 8 on both ranks (a documented option of Ash's recipe page, qwen3.8-flash-next.md lines 403 to 406: the same number on both ranks); the start line of the server reports 8 streams at once --server-ctx 262144 --server-ctx-source card box-3-flashnext-nvfp4-tp2-2x-tensorfold-1.2.5-official-1010 field ctx

How well it answers

Thinking off, except the last test. Seed 42. Sampling from the model card, with a presence penalty of 0 instead of 1.5 in the instruct tests (the server refuses 1.5; see the note below). A cut-off answer counts as wrong. Measured 2026-11-10.

Note from the run: MMLU-Pro (400 questions), GPQA Diamond (198), IFEval (200), the long-text tests at 8K and 32K tokens (RULER, 40 tasks each) and LiveCodeBench (40 tasks) ran with thinking off and the instruct sampling of the model card (temperature 0.7, top_p 0.8, top_k 20, min_p 0), with one forced change: a presence penalty of 0 instead of the card's 1.5, because the official server 1.2.5 refuses penalties above zero (see the speed section; it can change these numbers). Seed 42; the questions ran 8 at a time, the slot count of the server (same questions, limits and seed for every question). MMLU-Pro and GPQA Diamond ran with an answer limit of 48 tokens, letter only; IFEval with 4,096 tokens; the long-text tests with 128 tokens; LiveCodeBench with 32,768 tokens; the thinking test with 16,384 tokens. 1 of 400 MMLU-Pro answers was cut off at its limit and counts as wrong; no answer of GPQA Diamond, IFEval, the long-text tests or LiveCodeBench was cut off. The thinking test (30 GPQA Diamond questions, thinking on, temperature 1.0, top_p 0.95, top_k 20) cut off 8 of 30 answers at the limit of 16,384 tokens; these count as wrong, so the thinking number is a lower bound. The median thinking answer is 6,215 tokens long and takes 412.8 s (8 at a time, so the time is under load). LiveCodeBench answers are long even with thinking off: median 6,266 tokens, longest 24,617 (limit 32,768), and 1 of the 40 answers held no code. Long text by task: variable tracking 1 of 10 at 8K and 0 of 10 at 32K; multi-hop question answering 8 of 10 and 4 of 10. Speed v2, MMLU-Pro, GPQA Diamond, IFEval, the long-text tests and the thinking test ran one after the other on 10 Nov 2026 on one freshly started server; LiveCodeBench and the memory test each ran in their own session on a freshly started server. GPQA Diamond: the original is gated and no Hugging Face token was available, so the public mirror fingertap/GPQA-Diamond (198 rows) was used. These quality tests have no foreign-load guard; their sampling files list the busiest other process every 10 seconds: in the four instruct tests 18 of 210 samples were above 25 % CPU (up to 100 %), 13 of them ssh calls of my own watchdog script, 3 a firmware query of the box, 1 a package-manager check and 1 a short process listing (pgrep) of unknown origin; in the thinking test 8 of 206 samples (up to 100 %: 5 watchdog ssh calls, 2 firmware queries, 1 package-manager check); in LiveCodeBench 6 of 205 samples (up to 99 %: 3 of them the code runner of the test itself, 3 watchdog ssh calls). These are my own tests, and no other run of this setup is mixed into the numbers on this page.

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-nvfp4-tp2-2x-tensorfold-1.2.5-official-1010.json --parts mmlupro --job fn125c-1010

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-nvfp4-tp2-2x-tensorfold-1.2.5-official-1010.json --parts gpqa --job fn125c-1010

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-nvfp4-tp2-2x-tensorfold-1.2.5-official-1010.json --parts ifeval --job fn125c-1010

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-nvfp4-tp2-2x-tensorfold-1.2.5-official-1010.json --parts lcb --job fn125c-1010-lcb

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-nvfp4-tp2-2x-tensorfold-1.2.5-official-1010.json --parts ruler --job fn125c-1010

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-nvfp4-tp2-2x-tensorfold-1.2.5-official-1010.json --parts ruler --job fn125c-1010

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-nvfp4-tp2-2x-tensorfold-1.2.5-official-1010.json --parts think --job fn125c-1010

How much context fits

Memory test. The window is fixed at the engine's default (262,144 tokens), so the server is started once with the command of this setup and reserves its memory up front. Then, for each size (4K, 8K, 16K ... up to the largest size tested), one prompt that fills about 95 % of that size is sent, and the model must answer. The largest size that answered counts; it is the highest size tested, not a limit of the setup. Decode speed at that size was not measured. Peak memory is, on each of the two boxes, the rise of used memory (MemTotal - MemAvailable) over the idle value before the step; the table and the number show the highest of the two boxes. Unified memory, so GPU and CPU share it. Measured 2026-11-10. Raw files: result of every step · log · contract.

Note from the run: fit memory per node, highest rise over idle: box-3 67.2 GiB, box-4 61.7 GiB. The server was started once with the engine's default window of 262,144 tokens (8 requests at once, a prompt cache of 16.0 GiB) and reserves its memory at the start; the start line of rank 0 reads "model 42.47 GiB; 8 streams at once, 4.69 GiB each at a 262144-token window, of 54.0 GiB left after a 12.2 GiB reserve". Loading took 258.1 s. Then, for each size (4K, 8K, 16K, 32K, 64K, 128K and 256K), one prompt that fills about 95 % of the size was sent, and the model had to answer. All seven sizes answered; the largest size tested is 262,144 tokens, the whole window, so nothing larger exists on this server. Peak memory is, on each of the two boxes, the rise of used memory (MemTotal - MemAvailable, unified memory) over the idle value before the server start (8,926 and 9,447 MiB); the table and the number show the higher of the two boxes: 67.17 GiB on the first box and 61.67 GiB on the second at 262,144 tokens (47.83 GiB on the first box at 4K). The number includes the model weights. Decode speed at these sizes is not measured (the memory test returns no timings); the speed section has prefill and decode at 16K to 128K.

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-nvfp4-tp2-2x-tensorfold-1.2.5-official-1010.json --parts fit --job fn125c-1010-fit

How it plays the Gauntlet

One solo race, season seed 21, one attempt, no help.

Game rank as of Nov 10, 2026. The rank is the order of the Gauntlet finish times in this memory class. Race page and raw files.

The race ran the recipe named on its race page, with the sampling and answer limit of its lane config. The engine build of the race is not recorded in the race files.

Result (result.json) Race log (log.jsonl) Model answers (io.jsonl) System prompt Exact command Verify output Model config Race page

Also made this possible

• Engine (TensorFold official release v1.2.5, 8 Nov 2026)Ash Hart@ashxhart GitHub Hugging Face Website
• NVFP4 checkpoint (nvidia/Qwen3.8-Flash-Next-NVFP4, revision fc694b54)NVIDIAHugging Face GitHub Website
• Base model (Qwen3.8-Flash-Next)QwenHugging Face @Alibaba_Qwen GitHub Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

2x NVIDIA DGX Spark (GB10, 128 GB unified memory each), linked by CX7 RDMA over two links (109.28 and 109.18 Gb/s, measured before the run). Both run driver 580.178.04 and kernel 7.0.0-1019-nvidia, GPU clock capped at 2200 MHz. Box 3 is rank 0 and serves the HTTP API, box 4 is rank 1

Model

nvidia/Qwen3.8-Flash-Next-NVFP4 @fc694b54

NVFP4 checkpoint (nvidia/Qwen3.8-Flash-Next-NVFP4, revision fc694b54)

huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4

Base model (Qwen3.8-Flash-Next)

huggingface.co/Qwen/Qwen3.8-Flash-Next

Engine

Official release v1.2.5 (8 Nov 2026), tensor parallel over two boxes (--tp 2), MTP and copy drafts, 8 requests at once, 262,144 tokens of context, prompt cache 16 GiB. The author's documentation calls the two-box path experimental — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.5

Recipe (two-box NVFP4 path, docs/recipes/flash-next-cuda.md at v1.2.5)

github.com/ashhart/TensorFold/blob/v1.2.5/docs/recipes/flash-next-cuda.md by Ash Hart

Steps

1. Get the recipe, the engine and the model files from the list above.

2. Start rank 1 first, on the second box. It is the same command as in the next step with --rank 1, and without --name, --host and --port. Both boxes need the same TensorFold program, the same checkpoint files and NCCL 2.30 or later, and RANK0_ADDRESS is the address of the first box on the link between the boxes (set NCCL_SOCKET_IFNAME and NCCL_IB_HCA for that link, and LD_LIBRARY_PATH for NCCL). Each rank computes a digest of the whole checkpoint at its first start (about 2 minutes for this checkpoint), and both wait up to 30 minutes for the other (flash-next-cuda.md, lines 38 to 40).

   Copy

   tensorfold-native serve nvidia/Qwen3.8-Flash-Next-NVFP4 \
     --tp 2 \
     --rank 1 \
     --master RANK0_ADDRESS \
     --parallel 8

3. Then start rank 0, on the first box, with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8080, also from other machines in your network.

   Copy

   tensorfold-native serve nvidia/Qwen3.8-Flash-Next-NVFP4 \
     --tp 2 \
     --rank 0 \
     --master RANK0_ADDRESS \
     --parallel 8 \
     --name Qwen3.8-Flash-Next \
     --host 0.0.0.0 \
     --port 8080

   What the parts mean

   • --tp 2tensor parallel: the model is split across 2 ranks, here one rank on each box
   • --rank 0which of the two ranks this process is: rank 0 serves the HTTP API, rank 1 is started first
   • --master RANK0_ADDRESSthe address of rank 0, the first box; replace RANK0_ADDRESS with its address on the link between the boxes
   • --parallel 8how many requests it answers at the same time
   • --host 0.0.0.0who can connect: also other machines in your network. Careful: any machine in your network can then reach the server; use 127.0.0.1 if only this computer should
   • --port 8080the network port the server listens on
   • --name Qwen3.8-Flash-Nextthe model ID the server advertises to clients

TensorFold 1.2.5 official release archive tensorfold-1.2.5-linux-aarch64.tar.gz (sha256 1a0192b28d6ab6ec2749f3ff25515379dba4895171e781b30f810afc971ae4f8; tensorfold-native sha256 4f7ed2ab...), Ash Hart + contributors, NCCL 2.30.7 stock, TP=2 over 2 GB10 (sm_121), MTP + copy drafts, --parallel 8; tensorfold-native serve nvidia/Qwen3.8-Flash-Next-NVFP4 --tp 2 --rank 0 --master RANK0_ADDRESS --parallel 8 --name Qwen3.8-Flash-Next --host 0.0.0.0 --port 8080 (the command of rank 0, which serves the HTTP API; rank 1 is started first on the second box with the same command and --rank 1, without --name, --host and --port; environment on both boxes: LD_LIBRARY_PATH set to the folder of NCCL 2.30.7, NCCL_SOCKET_IFNAME and NCCL_IB_HCA set for the two CX7 links; RANK0_ADDRESS is the address of the first box on the CX7 link; our run used a start script that checks both boxes, starts rank 1, then rank 0 a few seconds later, and runs a watchdog)