Ash's TensorFold 1.2.2 on one DGX Spark: Qwen3.8-27B (EXL3 4.00 bpw)

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K and 64K tokens and of 130,048 tokens for the 128K step, which is reduced so that prompt plus 512 output tokens fit the 131,072 window (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-04. Raw files: speed result · speed contract · log · long answer.

The speed test ran with Ash's official release of TensorFold v1.2.2 (published 3 Nov 2026, 18:53 UTC): the prebuilt Linux/aarch64 archive from the release page, not a build of mine, on one DGX Spark, with the checkpoint and the draft model that Ash's measurements for this model use (turboderp/Qwen3.8-27B-exl3, branch 4.00bpw, revision 113cf7ab; z-lab/Qwen3.8-27B-DFlash2, revision 50307d4c) and his recipe's command: I ran the native engine's serve command with the EXL3 folder, the draft model given with --drafter, and --context 131072 (the recipe's own line is tensorfold serve with the EXL3 folder). Two points differ from the recipe text. The recipe's example shows the 3.00 bpw branch, but it sits in the section on the Python engine, and every measurement Ash gives for the native engine uses 4.00 bpw, so I used 4.00 bpw. The server default for this model is a context of 32,768 tokens, which would not take a 64K prompt, so I used the 131,072 of his measurements. I did not set the number of streams; the server chose 8 (--parallel auto), so the four concurrency levels (1, 2, 4 and 8 at once) are all inside its slot count. Sampling is the model card's instruct profile (temperature 0.7, top_p 0.8, top_k 20, presence penalty 1.5), seed 42, the request flag for no thinking; no cached tokens in the 61 requests that were checked. The headline 47.30 tok/s is the mean of the code prompt (48.98) and the prose prompt (45.61), medians of two runs each (spread 7.5 % and 3.7 %). Drafts were accepted at a rate of 19.7 % (1,527 of 7,766). Ash's release notes of v1.2.2 say: "Qwen3.8-27B EXL3 on a GB10: prompt prefill 38% to 42% faster from 2K to 64K tokens (494 to 705 tok/s at 2K), and decode 5% to 12% faster, with the same tokens (#1467)." Ash measures with his own bench tool (greedy decoding, a fresh server for every prompt length, prompt cache off); Speed v2 uses code and prose prompts with the model card's sampling and one server for all steps, so the percentages of the release notes and my numbers are not like-for-like. A like-for-like comparison I can give is on the same box with the same files, command and test: v1.2.1 (its own page) reaches 354.3 tok/s of prefill at 2K and 329.6 at 64K, v1.2.2 reaches 889.3 and 749.6 (factors 2.51 and 2.27); the first token of a 32K prompt comes after 95.4 s with v1.2.1 and after 39.7 s with v1.2.2; decode is 44.4 tok/s with v1.2.1 and 47.3 with v1.2.2; 8 requests at once give 60.9 tok/s in total with v1.2.1 and 108.8 with v1.2.2. The 8-at-once total is all tokens divided by wall time, so it includes the time to the first tokens of the 8 prompts. Decode at 16K tokens of context is 5.1 % (code prompt) and 12.9 % (prose prompt) higher with v1.2.2 than with v1.2.1; at 32K, 64K and 130K tokens it differs by at most 3.4 % in either direction. Each cell is the median of two runs, so I do not read small differences as a change. Ash's later comment on the pull request (2 Nov 2026) lists 340.4 to 798.8 tok/s for a 64K prompt with his tool. The load guard (a sample every half second) saw other processes together above 25 % CPU in 12 of 4,238 samples, at most 51.9 % (in that sample the Docker service used 49.9 %), never in two samples in a row, so it repeated no step. The ten-second quiet check saw at most 2.2 % CPU, and that was the test runner itself.

python3 speed2.py --base-url http://127.0.0.1:41409/v1 --label tf122-box-1-0411-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/tf122-box-1-0411-speed-r0/tf122-box-1-0411-speed-r0/speed2 --contract $RUN/raw/tf122-box-1-0411-speed-r0/tf122-box-1-0411-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/models/qwen38-27b-exl3-4.00bpw/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253 (same as the other Qwen3.8-27B cards); the server default (checkpoint generation_config.json: temperature 1.0, top_p 0.95, top_k 20) is overridden per request, which the API allows (api.md lines 174-179). Ash's command sets no sampling flag. Seed 42 for speed/quality. Same sampling as the Mac card of Nov 3; Ash's receipts are greedy (PR #1467), so his numbers are not sampled-comparable. --model qwen3.8-27b-exl3-4.00bpw-dflash2 --no-cache-prompt --server-log $RUN/logs/server-A.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 8 --server-slots-source TensorFold v1.2.2 README.md:97 and :326 (--parallel auto, the default the recipe command uses, asks for up to eight streams); server startup line 'model 17.07 GiB; 8 streams at once, 8.96 GiB each at a 131072-token window' (box-1 $HOME/tf122-0411/logs/probe-server-1.2.2-2000.log, probe start Nov 4, 2026 20:00); the recipe command sets no --parallel; C8 fits, no queue --server-ctx 131072 --server-ctx-source Ash's receipt PR #1467 ('--context 131072; greedy unless stated'); README.md v1.2.2 :291 says the server default for Qwen3.8-27B is 32,768 (takes up to 262,144); accepted value read from the server: startup line 'context: 131072' and GET /v1/models context_length 131072 / max_model_len 131072 (box-1 probe start Nov 4, 2026 20:00, logs/probe-server-1.2.2-2000.log)

Quality

Quality tests were not run on this setup: this pair of pages compares two releases of the same engine with the same weights and shows speed only. No quality numbers are shown here, and this setup has no rank.

Also made this possible

• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• EXL3 quant (4.00 bpw branch, revision 113cf7ab)turboderp@turboderp_ Hugging Face GitHub
• Draft model (DFlash2, revision 50307d4c)z-labHugging Face @zhijianliu_ GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA DGX Spark (GB10, 128 GB unified memory), one box. Driver 580.178.04, kernel 7.0.0-1019-nvidia, GPU clock capped at 2200 MHz.

Model

turboderp/Qwen3.8-27B-exl3 @113cf7ab (4.00bpw EXL3, dense 27B Gated DeltaNet + attention) + z-lab/Qwen3.8-27B-DFlash2 @50307d4c (drafter)

EXL3 quant (4.00 bpw branch, revision 113cf7ab)

huggingface.co/turboderp/Qwen3.8-27B-exl3/tree/113cf7ab958054860e43fb7f3063b1af19171095

Draft model (DFlash2, revision 50307d4c)

huggingface.co/z-lab/Qwen3.8-27B-DFlash2

Engine

Official release v1.2.2 of 3 Nov 2026, 18:53 UTC, the native CUDA engine, turboderp's EXL3 4.00 bpw checkpoint with DFlash2 drafts, 131,072 tokens of context, 8 streams — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.2

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8080.

   Copy

   tensorfold-native serve qwen38-27b-exl3-4.00bpw \
     --drafter qwen38-27b-dflash2 \
     --name qwen3.8-27b-exl3-4.00bpw-dflash2 \
     --port 8080 \
     --context 131072

   The file names are the ones we used. Run the command in the folder that holds them, or put the full path in front of each name.

   What the parts mean

   • --port 8080the network port the server listens on
   • --drafter qwen38-27b-dflash2the draft model: it guesses the next tokens and the main model checks them; without it the model decodes plain
   • --name qwen3.8-27b-exl3-4.00bpw-dflash2the model ID the server advertises to clients

TensorFold native (Zig) 1.2.2 official release archive tensorfold-1.2.2-linux-aarch64.tar.gz (sha256 17632c30), qwen3_5 CUDA family (dense EXL3), 1 GB10 (sm_121), DFlash2 drafter, --context 131072, server default streams; tensorfold-native serve $HOME/models/qwen38-27b-exl3-4.00bpw --drafter $HOME/models/qwen38-27b-dflash2 --name qwen3.8-27b-exl3-4.00bpw-dflash2 --port 8080 --context 131072 (recipe: tensorfold serve <EXL3 folder>; wrapper $HOME/tf122-0411/tf-ctl.sh)