Ash's TensorFold 1.2.7 with Mia's checkpoint: GLM-5.3-Flash on 2 GB10 boxes

How fast it runs

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer.

What ran. The speed test ran TensorFold 1.2.7 as the official release archive for Linux on arm64 (tensorfold-1.2.7-linux-aarch64.tar.gz, published 11 Nov 2026, 17:18 UTC by the repository owner, sha256 250b7cd3, the digest GitHub lists for that file; the binary tensorfold-native inside has the sha256 441a3847, recorded with the run), unchanged and without a patch, with the checkpoint that Mia publishes for it: Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold at revision 76c0b517 (a folder of the same name on both boxes, 83 shards; the revision file in it names the revision). It ran as two ranks, tensor parallel 2, started as the engine's recipe page for GLM-5.3-Flash on CUDA describes it (docs/recipes/glm-5.3-flash-cuda.md, section Run, lines 13 to 15 and 28: TF_GLM_RANK and TF_GLM_WORLD place each rank, NCCL_COMM_ID names rank 0's address and port, every rank gets the same arguments, NCCL_SOCKET_IFNAME and NCCL_IB_HCA name the link between the hosts). The command has no recipe of its own and no tuning: the engine's defaults plus --parallel 8 and --name GLM-5.3-Flash, no --context, no --drafter and no TF_GLM_DENSE_BITS (off by default, so the dense layers stay BF16: recipe page, line 88). The exact command is above.

The one change to the engine's setup: NCCL. The engine's runtime notes say that a GLM pair needs NCCL on both ranks and that it comes from the host (packaging/RUNTIME.md, lines 34 and 41: 'NVIDIA NCCL 2 runtime compatible with CUDA 13'); neither the package nor my two boxes carry it. I took NCCL 2.30.7 (the file libnccl.so.2.30.7 of the NVIDIA package libnccl2 2.30.7-1+cuda13.3) out of the Docker image that Mia's recipe 1104 builds, put the same file (sha256 fc7ea663, equal on both boxes) into a folder of my own on both boxes and pointed LD_LIBRARY_PATH at it, which the engine's own capabilities output names as the way; nothing was installed in the system. The start line of the engine reports NCCL 2.30.7 on rank 0 (recorded in the speed result). Besides this and my two flags, the command is the engine's own.

Window and streams. The window is 8,192 tokens, and the engine chose it, not I: I gave no --context. The README advises a positive limit for GLM (line 379: 'use a positive limit for GLM and inspect the capacity reported at startup'), and the recipe page's example gives --context 245760 with --parallel 1 (lines 21 and 22). The rank 0 start line recorded in the speed result reads '8 streams at once, the engine's own and 7 more taking 136 pages of 512 rows (0.85 GiB) as they need them (enough for 8 at a 8192-token window), 231 MiB fixed each, of 12.6 GiB left after a 12.2 GiB reserve' (the session of LiveCodeBench: 13.1 GiB left, the same window); the server's log also shows a prompt cache of 10.1 GiB (10.6 GiB in the LiveCodeBench session), which is not among the published files. GET /v1/models reports a context of 8192 (speed result). The KV cache is the engine's default for two ranks: pages in bf16 (the start line 'glm kv selection: pool pages ... cache bf16'; recipe page, line 56). With --parallel 8 the four levels of the concurrency test (1, 2, 4 and 8 at once) are all inside the streams. Both boxes carried their usual setting vm.min_free_kbytes = 1,048,576 during the run (the equality check before the run shows it; that file is not published); I changed nothing in the system for this run.

Long prompts (16K to 128K): no result. The window cannot hold them. The tool checked each size before it sent anything: a prompt of 16K, 32K, 64K or 128K tokens plus the 512 reply tokens is larger than the 8,192-token window that the server reports, so no request was sent and the four steps have no value (the register rows speed2_ctx_d16384_fits to speed2_ctx_d131072_fits are 0). The prefill series has one size, 2K, with a prompt of exactly 2,048 tokens in three runs (1,150, 1,150 and 1,151 tok/s); the sizes 8K and 32K were not run because a prompt of 8,192 tokens or more does not fit the window either.

Requests at once. The server has 8 streams, so all four levels (1, 2, 4 and 8 at once) are inside the streams. The total rises from 27.0 tok/s with 1 request to 57.7 tok/s with 8 requests, while the speed of one request falls from 27.0 to 7.4 tok/s. The words and characters per second come from the four decode runs.

The headline 28.0 tok/s is the mean of the code prompt (28.1) and the prose prompt (27.9), medians of two runs each (spread (CV) of 0.7 % and 1.8 %). The code prompt has 1,169 and 1,173 tokens and the prose prompt 1,290 and 1,291 tokens, and every reply is 512 tokens long and ends at the limit. Drafts were accepted at a rate of 50.1 % (1,140 of 2,276 drafted tokens, mean accepted length 2.26 tokens; counted from the engine's counters over the four decode runs).

Long answer test: no result. The test asks for one reply of up to 32,768 tokens to a short prompt of 161 tokens; the server refused the request before it answered (the refusal text is in the stream record of that request, which is not published; the output file l1-output.txt is empty), because the reply cap plus the prompt is larger than the window. The step is marked invalid, has no value and is not in the register.

Sampling. Temperature 1.0 and top_p 0.95 are the checkpoint's own generation_config.json (lines 9 and 10); the server's start line reads 'sampling: temperature 1.0, top_p 0.95', so the server uses them for a request without sampling fields. top_k 20 is not a server default (the server has none): the measuring tool sends it with every request, together with seed 42 and the flag for no thinking (enable_thinking false). Every speed request carries a nonce and a fresh cache salt, and the cache check of the tool found no cached tokens in any of the 38 requests that it checked.

Foreign load. The load guard of the speed test (any other process above 25 % of a core for two samples in a row, sampled twice a second; it watches the head box only) threw away and repeated no step: 0 of 16 steps, 859 samples in all. In three kept steps a single sample above 25 % was recorded (the system service snapd at 67.9 % and 34.0 %, the system-information helper landscape-sysin at 25.9 %); none lasted two samples in a row, so the guard did not repeat them. The quiet check of the speed run (one sample about every 10 seconds) took 43 samples, none above 25 % of a core; the highest was the run's own test runner at 3.1 %. The 10-second load files of both boxes (78 samples in all) show no foreign process above 25 % of a core. Not published: the guard's sample file, the quiet-check tables, the load files and the single requests of the speed run.

python3 speed2.py --base-url http://127.0.0.1:39305/v1 --label glm-stock127-2x-1112-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/glm-stock127-2x-1112-speed-r0/glm-stock127-2x-1112-speed-r0/speed2 --contract $RUN/glm-stock127-2x-1112-speed-r0/glm-stock127-2x-1112-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/models/glm53-mia-4bpw-1104/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 1.0, "top_p": 0.95, "top_k": 20} --sampling-source Sampling of the card: temperature 1.0, top_p 0.95, top_k 20, sent in every request by the tool. Source of T/top_p: checkpoint generation_config.json:9-10 (Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold, revision 76c0b517), which the stock server uses for requests without sampling fields (start line 'sampling: temperature 1.0, top_p 0.95'). top_k 20: not a stock default (the stock server has none); the tool sends it in every request. Speed v2 runs with thinking off (enable_thinking false per request), think part with thinking on. --model GLM-5.3-Flash --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 8 --server-slots-source both ranks started with --parallel 8; rank 0 start line: '8 streams at once' (load probe Nov 12, 02:06) --server-ctx 8192 --server-ctx-source rank 0 start line of the unchanged TensorFold 1.2.7 at --parallel 8: 'serving GLM-5.3-Flash ... context: 8192' (no --context given; the engine chose the window itself; load probe Nov 12, 02:06; the recipe page's example gives --context 245760, and README line 379 advises a positive limit for GLM)

How well it answers

Three tests have a result. GPQA with 30 questions and thinking and LiveCodeBench have none: the server refused every one of their requests, because the reply cap plus the prompt does not fit the 8,192-token window, so no percentage and no margin is shown for them.

Thinking off, except the last test. Seed 42. Sampling from the checkpoint's own settings plus top_k 20 (see the notes). A cut-off answer counts as wrong. Measured 2026-11-12.

Note from the run: MMLU-Pro and GPQA Diamond ran with an answer limit of 1,024 tokens (the tool's default is 48; the card of this setup sets 1,024; thinking off; the questions, the seed 42 and the first-letter extraction are the usual ones) and IFEval with 4,096 tokens (prompt-level strict). Cut off at their limits: 0 of 400 MMLU-Pro, 2 of 198 GPQA and 8 of 200 IFEval answers; a cut-off answer counts as wrong. The server refused none of these 798 requests. The questions of each test ran 8 at a time, the 8 streams of the server (same questions, limits and seed for every question). GPQA Diamond ran from the public mirror fingertap/GPQA-Diamond (198 questions), because the original is gated. The percentages are computed from the counts (GPQA Diamond: 89 of 198 is 44.9 %; the summary file shows 44.95, which is already rounded to two decimals).

No result: GPQA with 30 questions and thinking, and LiveCodeBench. The server refused every request of these two tests before it answered: all 30 of the thinking test (reply cap 16,384 tokens) and all 40 of LiveCodeBench (reply cap 32,768 tokens), with HTTP 400 and the text 'maximum context length is 8192 tokens' (every one of the 70 answer records of the two answer files carries it; the prompts had 95 to 1,303 tokens). The reply cap plus the prompt is larger than the 8,192-token window. The raw files of these two tests (summary, contract and log) carry 0.0 % and a margin, because the measuring tool counts a refused request as wrong; that is not a measurement of the model: every request was refused and none was answered. The page shows no percentage and no margin for these two tests, and they are in no mean and no tile. The median time per question of the thinking test (0.01 s) is the time to the refusal, not a speed value. I changed no list and no cap. The engine's own log of rank 0 agrees with the raw records: it lists 71 refused requests in the sessions of this page (30 with a reply cap of 16,384 tokens and 41 with 32,768), which are the 30 + 40 refusals of these two tests and the refused request of the long answer test; that log is not published.

Not run: the long-text tests (RULER-lite at 8K and 32K). Their prompts have exactly 8,192 and 32,768 tokens, which do not fit a window of 8,192 tokens together with a reply, so they were not started; they are not counted as wrong.

Sampling. The tests sent temperature 1.0, top_p 0.95 and top_k 20 with every request (the sources are in the speed notes), seed 42, cache_prompt false and the flag for no thinking (enable_thinking false) in the three instruct tests; the thinking test asked for thinking. One run of each test, no repetition.

Foreign load. These tests have no load guard; their quiet check lists the busiest other process about every 10 seconds. The three instruct tests: 266 samples, 2 above 25 % of a core (a package-manager check at 96.3 % and a firmware query at 58.3 %, each in one sample); the thinking test and LiveCodeBench: 1 sample each, at 0.3 %. Nothing was repeated. Not published: the quiet-check tables; the MMLU-Pro answer file, because one answer repeats a figure from an engineering question that is written with a power unit, which the site's filter for power figures holds back (it is not a measurement); the other summaries, logs, contracts and answer files that the site's filter lets through are published. Times (box time, 12 Nov 2026): Speed v2 from 02:18 to 02:25, the three instruct tests from 02:26 to 03:10, the thinking test at 03:10, LiveCodeBench at 03:12.

bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-tf-127-stock-2x-mia-76c0b517-1112.json --parts mmlupro --job glm-stock127-2x-1112

bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-tf-127-stock-2x-mia-76c0b517-1112.json --parts gpqa --job glm-stock127-2x-1112

bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-tf-127-stock-2x-mia-76c0b517-1112.json --parts ifeval --job glm-stock127-2x-1112

bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-tf-127-stock-2x-mia-76c0b517-1112.json --parts lcb --job glm-stock127-2x-1112-lcb

bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-tf-127-stock-2x-mia-76c0b517-1112.json --parts think --job glm-stock127-2x-1112

Also made this possible

• ModelZ.ai (GLM-5.3-Flash)Hugging Face @Zai_org GitHub Website
• Engine (TensorFold official release v1.2.7, 11 Nov 2026, 17:18 UTC; recipe docs/recipes/glm-5.3-flash-cuda.md)Ash Hart@ashxhart GitHub Hugging Face Website
• EXL3 formatturboderp@turboderp_ Hugging Face GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

2x NVIDIA DGX Spark (GB10), tensor parallel 2 over a direct CX7 RDMA link (two paths, 109.3 Gb/s each, measured before the run): one Gigabyte AI TOP ATOM (rank 0, the head box that serves HTTP) and one Lenovo ThinkStation PGX (rank 1). Both run driver 580.178.04 and kernel 7.0.0-1019-nvidia, GPU clock capped at 2200 MHz

Model

GLM-5.3-Flash (Z.ai), EXL3 4bpw by Mia-AiLab (rev 76c0b517), no DFlash2 (no --drafter flag set)

Checkpoint (EXL3 4 bpw, revision 76c0b517, Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold)

huggingface.co/Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold

Engine

TensorFold 1.2.7, the official release package for Linux (aarch64), unchanged; Mia's EXL3 4 bpw checkpoint (revision 76c0b517); tensor parallel on 2 GB10 boxes over CX7 RDMA; window 8,192 tokens and 8 streams, with the engine's own choices apart from --parallel 8 — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.7

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory.

   Copy

   env LD_LIBRARY_PATH=<folder with libnccl.so.2.30.7> \
     TF_GLM_WORLD=2 \
     NCCL_COMM_ID=<address of rank 0>:29583 \
     NCCL_SOCKET_IFNAME=<network interface of the link> \
     NCCL_IB_HCA=<RDMA device of the link> \
     TF_GLM_RANK=<1|0> tensorfold-native serve <folder of Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold at revision 76c0b517> \
     --name GLM-5.3-Flash \
     --parallel 8

   What the parts mean

   • --parallel 8how many requests it answers at the same time
   • --name GLM-5.3-Flashthe model ID the server advertises to clients

TensorFold 1.2.7, official package, unchanged (ashhart/TensorFold v1.2.7, Zig engine tensorfold-native, TP=2, MTP drafts + copies); the only deviation: NCCL 2.30.7 from the Docker image that Mia's recipe 1104 builds (libnccl2 2.30.7-1+cuda13.3), via LD_LIBRARY_PATH; env LD_LIBRARY_PATH=<folder with libnccl.so.2.30.7> TF_GLM_WORLD=2 NCCL_COMM_ID=<address of rank 0>:29583 NCCL_SOCKET_IFNAME=<network interface of the link> NCCL_IB_HCA=<RDMA device of the link> TF_GLM_RANK=<1|0> tensorfold-native serve <folder of Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold at revision 76c0b517> --name GLM-5.3-Flash --parallel 8 (rank 1 first, then rank 0, the same arguments on both ranks; the engine's recipe page docs/recipes/glm-5.3-flash-cuda.md, section Run, gives the same variables, lines 13 to 15 and 28, and shows NVIDIA's NVFP4 checkpoint with --context 245760 --parallel 1 as its example. I used Mia's checkpoint from a local folder on both boxes in place of a Hugging Face name, added --parallel 8 and --name, and gave no --context)