Ash's TensorFold 1.2.1 on one DGX Spark: Qwen3.8-27B (EXL3 4.00 bpw)

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K and 64K tokens and of 130,048 tokens for the 128K step, which is reduced so that prompt plus 512 output tokens fit the 131,072 window (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-04. Raw files: speed result · speed contract · log · long answer.

The speed test ran with Ash's official release of TensorFold v1.2.1 (published 1 Nov 2026, 23:52 UTC): the prebuilt Linux/aarch64 archive from the release page, not a build of mine, on one DGX Spark, with the checkpoint and the draft model that Ash's measurements for this model use (turboderp/Qwen3.8-27B-exl3, branch 4.00bpw, revision 113cf7ab; z-lab/Qwen3.8-27B-DFlash2, revision 50307d4c) and his recipe's command: I ran the native engine's serve command with the EXL3 folder, the draft model given with --drafter, and --context 131072 (the recipe's own line is tensorfold serve with the EXL3 folder). Two points differ from the recipe text. The recipe's example shows the 3.00 bpw branch, but it sits in the section on the Python engine, and every measurement Ash gives for the native engine uses 4.00 bpw, so I used 4.00 bpw. The server default for this model is a context of 32,768 tokens, which would not take a 64K prompt, so I used the 131,072 of his measurements. I did not set the number of streams; the server chose 8 (--parallel auto), so the four concurrency levels (1, 2, 4 and 8 at once) are all inside its slot count. Sampling is the model card's instruct profile (temperature 0.7, top_p 0.8, top_k 20, presence penalty 1.5), seed 42, the request flag for no thinking; no cached tokens in the 61 requests that were checked. The headline 44.36 tok/s is the mean of the code prompt (45.86) and the prose prompt (42.87), medians of two runs each (spread 3.0 % and 2.5 %). Drafts were accepted at a rate of 18.0 % (1,491 of 8,264). This page is the control run for the page of v1.2.2: the same box, the same weights, the same command and the same test, only the engine release differs. It ran on the same evening, after the v1.2.2 run. The prefill series of pull request #1467 is in v1.2.2 only; the series of #1465 is in v1.2.0 and v1.2.1. Prefill at 2K is 354.3 tok/s here and 889.3 tok/s with v1.2.2; the first token of a 32K prompt comes after 95.4 s here and 39.7 s with v1.2.2; decode is 44.4 tok/s here and 47.3 with v1.2.2. The load guard (a sample every half second) saw other processes together above 25 % CPU in 103 of 10,244 samples, up to 116 %, including the samples of the three tries that the guard threw away. The guard throws a try away and repeats it when a single process uses more than 25 % CPU in two samples in a row. The load guard threw away and repeated 3 tries: the long single run once, because of a helper of the snap package system (46 % CPU), and the prose run of the 130K prompt twice, because of "apt update" and of the firmware service fwupd (up to 100 % CPU each); every value kept is from a clean try according to the guard.

python3 speed2.py --base-url http://127.0.0.1:39017/v1 --label tf121-box-1-0411-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/tf121-box-1-0411-speed-r0/tf121-box-1-0411-speed-r0/speed2 --contract $RUN/raw/tf121-box-1-0411-speed-r0/tf121-box-1-0411-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/models/qwen38-27b-exl3-4.00bpw/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253 (same as the other Qwen3.8-27B cards); the server default (checkpoint generation_config.json: temperature 1.0, top_p 0.95, top_k 20) is overridden per request, which the API allows (api.md lines 174-179). Ash's command sets no sampling flag. Seed 42 for speed/quality. Same sampling as the Mac card of Nov 3; Ash's receipts are greedy (PR #1467), so his numbers are not sampled-comparable. --model qwen3.8-27b-exl3-4.00bpw-dflash2 --no-cache-prompt --server-log $RUN/logs/server-A.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 8 --server-slots-source TensorFold v1.2.1 README.md (same text as v1.2.2 README.md:97 and :326, --parallel auto asks for up to eight streams); server startup line 'model 17.07 GiB; 8 streams at once, 8.90 GiB each at a 131072-token window' (box-1 $HOME/tf122-0411/logs/probe-server-1.2.1-2043.log, probe start Nov 4, 2026 20:43); the recipe command sets no --parallel; C8 fits, no queue --server-ctx 131072 --server-ctx-source Ash's receipt PR #1467 ('--context 131072; greedy unless stated'); the server default for Qwen3.8-27B is 32,768 (v1.2.2 README.md:291, v1.2.1 README.md:282 per the Nov 3 card); accepted value read from the server: startup line 'context: 131072' and GET /v1/models context_length 131072 (box-1 probe start Nov 4, 2026 20:43, logs/probe-server-1.2.1-2043.log)

Quality

Quality tests were not run on this setup: this pair of pages compares two releases of the same engine with the same weights and shows speed only. No quality numbers are shown here, and this setup has no rank.

Also made this possible

• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• EXL3 quant (4.00 bpw branch, revision 113cf7ab)turboderp@turboderp_ Hugging Face GitHub
• Draft model (DFlash2, revision 50307d4c)z-labHugging Face @zhijianliu_ GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA DGX Spark (GB10, 128 GB unified memory), one box. Driver 580.178.04, kernel 7.0.0-1019-nvidia, GPU clock capped at 2200 MHz.

Model

turboderp/Qwen3.8-27B-exl3 @113cf7ab (4.00bpw EXL3, dense 27B Gated DeltaNet + attention) + z-lab/Qwen3.8-27B-DFlash2 @50307d4c (drafter)

EXL3 quant (4.00 bpw branch, revision 113cf7ab)

huggingface.co/turboderp/Qwen3.8-27B-exl3/tree/113cf7ab958054860e43fb7f3063b1af19171095

Draft model (DFlash2, revision 50307d4c)

huggingface.co/z-lab/Qwen3.8-27B-DFlash2

Engine

Official release v1.2.1 of 1 Nov 2026, 23:52 UTC, the native CUDA engine, turboderp's EXL3 4.00 bpw checkpoint with DFlash2 drafts, 131,072 tokens of context, 8 streams — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.1

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8080.

   Copy

   tensorfold-native serve qwen38-27b-exl3-4.00bpw \
     --drafter qwen38-27b-dflash2 \
     --name qwen3.8-27b-exl3-4.00bpw-dflash2 \
     --port 8080 \
     --context 131072

   The file names are the ones we used. Run the command in the folder that holds them, or put the full path in front of each name.

   What the parts mean

   • --port 8080the network port the server listens on
   • --drafter qwen38-27b-dflash2the draft model: it guesses the next tokens and the main model checks them; without it the model decodes plain
   • --name qwen3.8-27b-exl3-4.00bpw-dflash2the model ID the server advertises to clients

TensorFold native (Zig) 1.2.1 official release archive tensorfold-1.2.1-linux-aarch64.tar.gz (sha256 74a1047d), qwen3_5 CUDA family (dense EXL3), 1 GB10 (sm_121), DFlash2 drafter, --context 131072, server default streams; tensorfold-native serve $HOME/models/qwen38-27b-exl3-4.00bpw --drafter $HOME/models/qwen38-27b-dflash2 --name qwen3.8-27b-exl3-4.00bpw-dflash2 --port 8080 --context 131072 (recipe: tensorfold serve <EXL3 folder>; wrapper $HOME/tf122-0411/tf-ctl.sh)