Mia's 2-Spark recipe: GLM-5.3-Flash
2x NVIDIA GB10 boxes linked by CX7 RDMA (about 109 Gb/s): 1x Gigabyte AI TOP ATOM and 1x Lenovo ThinkStation PGX. Both run driver 580.178.04 and kernel 7.0.0-1019-nvidia, GPU clock capped at 2200 MHz.

Overall score as of 23 Oct 2026. Scores are relative to the best run in the class today; a better model later moves older runs down.
How the overall score is calculated
| Part | This run | Best in class | Relative | Weight | Points |
|---|---|---|---|---|---|
| Speed, one request | decode 49.9 tok/s | best 68.5 tok/s | 72.9 % | 17.5 % | 12.8 |
| Speed, 8 at once | total 99.2 tok/s | best 99.6 tok/s | 99.7 % | 17.5 % | 17.4 |
| Quality | mean of 7 tests: 62.4 % | best mean 69.1 % | 90.2 % | 25 % | 22.6 |
| Browser check | 9 of 10 | 90.0 % | 5 % | 4.5 | |
| My rating | 5 of 10 | 50.0 % | 35 % | 17.5 | |
| Overall | 100 % | 74.7 |
The seven quality tests: MMLU-Pro 64.8 · GPQA 36.9 · IFEval 79.0 · LiveCodeBench 52.5 · long text 8K 75.0 · long text 32K 55.0 · GPQA 30 with thinking 73.3.
The speed part is half the decode speed with one request and half the total speed with 8 requests at once, each relative to the best run of this class; a run without a total for 8 requests at once gets the better of its decode speed alone and the mix with its highest measured total. Quality is relative to the best run of this class, the browser check and my rating are out of 10. The parts are rounded one by one. sparkDash, the memory test and speculative-decoding numbers are not part of it. How the score is made, and the assumption behind it.
What it builds as an agent
Task: Voxel world (3D scene, one HTML file). One attempt, no help. Counted: the final file counts.
The exact promptcopy it and try it yourself↓
This is the task prompt word for word, as we handed it to the agent harness. The harness adds its own system prompt and tools, and some harnesses add short context notes of their own. A run that goes past 90 minutes also gets a short “finish now” message, as the prompt says. To try it yourself, give it to your own coding agent: it needs a shell and a working folder.
Design and create the most impressive, creative, elaborate and detailed
voxel art world you are capable of. The theme and style are ENTIRELY your
choice — pick whatever world you can execute best and push it as far as
you can: scale, variety, detail, color, animation, atmosphere, life.
Surprise me.
Hard requirements:
- Everything ends up in ONE single self-contained HTML file (game.html in
the current working directory) that opens and runs directly in Chrome.
Use WebGL and whatever libraries you like, but embed everything —
no internet access at view time, no external files.
- The scene must present itself: include an automatic slow cinematic
camera tour (orbiting the world, occasionally zooming in on highlights)
that starts on load, plus mouse controls (orbit/zoom) for manual viewing.
- All user-facing text (UI, HUD, titles, descriptions, hints) must be in
ENGLISH, regardless of the language of any project files you encounter.
When finished, verify your HTML actually loads and renders without
console errors, then state DONE with the absolute path of game.html.
Do not ask questions.
How you work (same for every model):
- You work alone in the folder /work (this is the current working directory). Nobody will answer questions. Aim to finish in about 60 minutes. After 90 minutes you will get a message to finish; then you have at most 30 more minutes. Hard stop after 120 minutes.
- Tools on this machine: shell, files, python3 with numpy and Pillow, ffmpeg, node, and `shot.sh game.html` (opens the page in a headless browser, saves a screenshot and prints console errors). There is no internet and no access to other machines.
- Keep a working /work/game.html at all times. If you are stopped, the last version that loaded without errors counts.
- JavaScript libraries available offline in the test browser, through an import map with these exact URLs: "three": "https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.module.js", "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/", "simplex-noise": "https://cdn.jsdelivr.net/npm/simplex-noise@4.0.3/dist/esm/simplex-noise.js", "@tweenjs/tween.js": "https://cdn.jsdelivr.net/npm/@tweenjs/tween.js@25.0.0/dist/tween.esm.js", "cannon-es": "https://cdn.jsdelivr.net/npm/cannon-es@0.20.0/dist/cannon-es.js". No other external resources.
- After the scene is built, set `window.scene` to the Three.js scene (if you use Three.js) and `window.ready = true`.Play it yourself
Open full screenAI-generated The page this setup built, running live in your browser: click into it and use the controls it shows itself (usually drag to look around, scroll to zoom). It needs WebGL and can be slow on a weak graphics card.
The browser check
- page loads without page error1 / 1
- window.ready === true within 30 s1 / 1
- no console errors (whole session)1 / 1
- frame is not blank1 / 1
- frame rate in software GL (rAF over 5 s)0 / 1
- world: scene size 0.5 (parts >= 80 or >= 20000 triangles; canvas fallback), visible text English if any 0.51 / 1
- automatic camera tour: the view changes on its own in two successive ~5 s windows (1 each)2 / 2
- animation in the world (nodes / instances / particles moving besides the camera)1 / 1
- OrbitControls: drag and wheel change the picture beyond the animation baseline1 / 1
Measured 2026-10-23. Raw files: check · run info · command · final choice · contract · sampling · gate stats · gate log · screenshot · clip.mp4.
Browser check 9.0 out of 10 (voxel-world task): the page loads, runs without console errors, is not blank, shows a world (48,679 parts, 584,652 triangles, English text), flies an automatic camera tour in both windows, animates (129 animated nodes) and answers to mouse drag and wheel; the one lost point is the frame rate (0.98 frames per second in software rendering without a GPU, 20 are needed). The agent (OpenCode 1.19.3 with my launcher that sets the model's input limit to its context; no harness with a fixed version and sampling is named for local use by Mia's recipe or by the model maker, so OpenCode) finished by itself after 71.1 minutes, one try, no help; the “finish now” message (from 90 minutes) was not needed. 14 versions of the page were seen, all 14 loaded without errors, and the last one counts. 58 requests, no error, no request blocked (6,110,469 prompt tokens, 109,305 answer tokens); 56 requests ended with a tool call and 2 with a normal stop, none at the limit; the longest answer had 54,782 tokens. The limit per answer was 65,536 tokens, the value the model maker's page names for its Terminal-Bench 2.1 and NL2Repo agent results (the page names a larger one, 163,840 tokens, for a different test, HLE with tools); none of the 58 answers reached the limit. The recipe's server default of 32,768 tokens applies only when a request sets none, so I raised the limit for this test; the server setup is unchanged. Thinking was on with the recipe's sampling (temperature 1.0, top_p 0.95), the context was 1,048,576 tokens. The agent container ran on the second Spark.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
run_showcase.sh --harness opencode-limit-input --task VX --upstream http://127.0.0.1:18080/v1 --model GLM-5.3-Flash-EXL3 --sampling-file $RUN/raw/glm-py-2310-show-r5/show.in/sampling-show.json --out $RUN/raw/glm-py-2310-show-r5/show --limit-s 7200 --inject-after-s 5400 --poll-s 20 --ctx 1048576How fast it runs
| Measurement | Result |
|---|---|
| Decode | 49.9 tok/s |
| Words per second | 35.8 words/s |
| Characters per second | 230.4 chars/s |
| Decode, code prompt | 51.3 tok/s |
| Decode, prose prompt | 48.6 tok/s |
| Spread of the decode runs (CV) | 2.6 % |
| Spread of the code runs (CV) | 2.6 % |
| Spread of the prose runs (CV) | 2.6 % |
| Decode, window 1 | 46.3 tok/s |
| Prefill at 2K | 1798.8 tok/s |
| Prefill at 8K | 1916.8 tok/s |
| Prefill at 32K | 1943.5 tok/s |
| Time to first token at 2K | 1.1 s |
| Time to first token at 8K | 4.3 s |
| Time to first token at 32K | 16.9 s |
| Draft acceptance rate | 61.7 % |
| Mean accepted length | 2.3 tokens |
| 1 request at once (total) | 46.6 tok/s |
| 2 at once (total) | 59.2 tok/s |
| 4 at once (total) | 80.2 tok/s |
| 8 at once (total) | 99.2 tok/s |
| 1 request at once (per request) | 46.6 tok/s |
| 2 at once (per request) | 30.5 tok/s |
| 4 at once (per request) | 20.4 tok/s |
| 8 at once (per request) | 12.7 tok/s |
| Server slots (from the recipe) | 8 slots |
| Long answer, wall time | 73.2 s |
Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-10-23. Raw files: speed result · speed contract · long answer.
Requests at once: the server has 8 slots (from the author's recipe). The speed test ran with the author's recipe (MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold, commit 3fd60dfc, the head of main on 23 Oct 2026): Mia's Python engine (TensorFold v0.6.0 with her 107 patches, in the image the recipe pins), tensor parallel over two GB10 boxes (CX7 RDMA, about 109 Gb/s), DFlash2 drafts and vision on as the recipe sets them, a context window of 1,048,576 tokens and 8 requests at once (PARALLEL=8). The recipe's default on two GB10 boxes is 4 requests at once and it documents up to 8; I took the largest setting the server allows at full context, as my test method says, and name it here. Sampling is the one in the recipe and in the model's generation config, the same with thinking on and off: temperature 1.0, top_p 0.95; seed 42; the request flag for no thinking; no cached tokens in the 45 requests that were checked. The headline 49.94 tok/s is the mean of the code prompt (51.25) and the prose prompt (48.64), medians of two runs each (spread 2.6 % and 2.6 %). Drafts were accepted at a rate of 61.7 % (1,168 of 1,893). The server has 8 slots, so the four concurrency levels (1, 2, 4 and 8 at once) are all inside its slot count. This page is the Python engine, Mia's current standard recipe for this model. Her Zig engine (another branch) is a different setup and is not mixed with it.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 speed2.py --base-url http://127.0.0.1:44455/v1 --label glm-py-2310-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/glm-py-2310-speed-r0/glm-py-2310-speed-r0/speed2 --contract $RUN/raw/glm-py-2310-speed-r0/glm-py-2310-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1 --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --tokenizer-json $HOME/.cache/huggingface/hub/models--Mia-AiLab--GLM-5.3-Flash-EXL3-4bpw-TensorFold/snapshots/078455ffe6472f9a52fbc1139f58b9db2881b25c/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 1.0, "top_p": 0.95} --sampling-source Same setting for thinking and non-thinking, no separate profile: Mia's recipe README.md:577 @3fd60dfc ('The checkpoint's own sampling defaults apply (temperature 1.0, top_p 0.95)'), scripts/config.sh:243-250 (sampled decode with the checkpoint's defaults, NUCLEUS_UNION=1: 'sampling at greedy's pace'), checkpoint generation_config.json:9-10 (temperature 1.0, top_p 0.95; Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold @078455ff, identical to zai-org/GLM-5.3-Flash generation_config.json:9-10, copy not published), zai-org/GLM-5.3-Flash README.md:45 (thinking via reasoning_effort, default max) and :50 (eval T=1.0, top_p=0.95). Speed v2 runs with thinking off (enable_thinking false per request), think/show with thinking on (server default THINKING=1, config.sh:131). --model GLM-5.3-Flash-EXL3 --no-cache-prompt --server-log $RUN/logs/server-A.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 8 --server-slots-source MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold@3fd60dfc (main = Mia's page v1.10) scripts/config.sh:104-110 (PARALLEL 1 to 8 with drafter=dflash2; default 4 on two GB10 boxes), README.md:49 ('Up to 8 requests at once (PARALLEL=8, v1.5)'), :506 (table), start.sh:60 (--parallel $PARALLEL); SENDEPLAN 7.3 (Niklas Oct 22 23:0x): the largest setting the author's server allows at full context (README.md:57-62: pool ~1.5M tokens at 8, so the 1M window fits), named openly; the start log must show 'Window 1048576 tokens · 8 at once'How well it answers
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| MMLU-Pro 400 fixed questions, letter only | 64.8 % ± 4.7 pts | 259 / 400 | 1 | summary · log · contract | command |
| GPQA Diamond 198 questions, letter only | 36.9 % ± 6.7 pts | 73 / 198 | 17 | answers · summary · log · contract | command |
| IFEval 200 fixed prompts, strict | 79.0 % ± 5.6 pts | 158 / 200 | 0 | summary · log · contract | command |
| LiveCodeBench 40 newest tasks, the code runs | 52.5 % ± 14.8 pts | 21 / 40 | 19 | answers · summary · log · contract | command |
| Long text 8K 4 tasks x 10 at 8192 tokens | 75.0 % ± 13.0 pts | 30 / 40 | 8 | log · contract · answers · summary | command |
| Long text 32K 4 tasks x 10 at 32768 tokens | 55.0 % ± 14.7 pts | 22 / 40 | 9 | log · contract · answers · summary | command |
| GPQA 30, thinking on 30 questions, limit 16,384 tokens | 73.3 % ± 15.1 pts median 4,199 tokens, 271.4 s per question | 22 / 30 | 7 | log · contract · summary | command |
Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-10-23.
Note from the run: MMLU-Pro and GPQA Diamond ran with an answer limit of 1,024 tokens (my usual limit for these two tests is 48; the card of this setup sets 1,024); same questions, seed 42, first-letter extraction. 1 of 400 MMLU-Pro answers and 17 of 198 GPQA Diamond answers were still cut off at that limit and count as wrong. 19 of 40 LiveCodeBench answers were cut off at their limit of 32,768 tokens, 17 of 80 long-text answers at their limit of 128 tokens (8 at 8K and 9 at 32K) and 7 of 30 GPQA answers with thinking at the limit of 16,384 tokens; these count as wrong too. The questions of each test ran 8 at a time, the slot count of the recipe's server (same questions, limits and seed for every question). Sampling: temperature 1.0, top_p 0.95 (the recipe and the model's generation config). These are my own tests, and no other run of this setup is mixed into the numbers on this page.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-exl3-4bpw-python-3fd60dfc-2x.json --parts mmlupro --job glm-py-2310bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-exl3-4bpw-python-3fd60dfc-2x.json --parts gpqa --job glm-py-2310bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-exl3-4bpw-python-3fd60dfc-2x.json --parts ifeval --job glm-py-2310bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-exl3-4bpw-python-3fd60dfc-2x.json --parts lcb --job glm-py-2310bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-exl3-4bpw-python-3fd60dfc-2x.json --parts ruler --job glm-py-2310bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-exl3-4bpw-python-3fd60dfc-2x.json --parts ruler --job glm-py-2310bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-exl3-4bpw-python-3fd60dfc-2x.json --parts think --job glm-py-2310How much context fits
| Context size | Prompttokens | Filled | First token | Prefilltok/s, as the server reports it | Decodetok/s | Peak memoryGiB | Result |
|---|---|---|---|---|---|---|---|
| 4K 4,096 tokens | 3,892 | 95.0 % | — | — | — | 99.47 | answered (stop) |
| 8K 8,192 tokens | 7,782 | 95.0 % | — | — | — | 100.38 | answered (length) |
| 16K 16,384 tokens | 15,564 | 95.0 % | — | — | — | 100.39 | answered (stop) |
| 32K 32,768 tokens | 31,129 | 95.0 % | — | — | — | 99.76 | answered (length) |
| 64K 65,536 tokens | 62,259 | 95.0 % | — | — | — | 99.85 | answered (stop) |
| 128K 131,072 tokens | 124,518 | 95.0 % | — | — | — | 100.03 | answered (stop) |
Memory test. The recipe fixes the context (1,048,576 tokens), so the server is started once with the recipe's own command and reserves its memory up front. Then, for each size (4K, 8K, 16K ... up to the largest size tested), one prompt that fills about 95 % of that size is sent, and the model must answer. The largest size that answered counts; it is the highest size tested, not a limit of the setup. Decode speed at that size was not measured. Peak memory is, on each of the two boxes, the rise of used memory (MemTotal - MemAvailable) over the idle value before the step; the table and the number show the highest of the two boxes. Unified memory, so GPU and CPU share it. Measured 2026-10-23. Raw files: result of every step · log · contract.
Note from the run: fit memory per node, highest rise over idle: box-3 100.4 GiB, box-4 99.6 GiB The recipe fixes the context (1,048,576 tokens, 8 requests at once), so the server is started once with the author's command and reserves its memory up front. Every size from 4K to 128K answered. The largest size tested is 131,072 tokens, the highest size in my list, not a limit of the setup: the window is 1,048,576. Decode speed at these sizes is not measured (the server returns no timings). The memory number is the rise of used memory over the idle value (unified memory, so GPU and CPU share it). The number shown (100.03 GiB) is the rise on the first GB10 box at the largest size; the highest rise at any size was 100.39 GiB on the first and 99.59 GiB on the second Spark.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/glm53-flash-exl3-4bpw-python-3fd60dfc-2x.json --parts fit --job glm-py-2310Also made this possible
- Model
Z.ai (GLM-5.3-Flash)Hugging Face @Zai_org GitHub Website - Draft model (DFlash2)
Inco AI@inco_ai Hugging Face GitHub Website - Engine (TensorFold)
Ash Hart@ashxhart GitHub Hugging Face Website - EXL3 format
turboderp@turboderp_ Hugging Face GitHub
Run it yourself
What you need and how to start it, step by step. The results below come from exactly this setup.
What you need
- Hardware
- 2x NVIDIA GB10 boxes linked by CX7 RDMA (about 109 Gb/s): 1x Gigabyte AI TOP ATOM and 1x Lenovo ThinkStation PGX. Both run driver 580.178.04 and kernel 7.0.0-1019-nvidia, GPU clock capped at 2200 MHz.
- Model
- Mia GLM-5.3-Flash EXL3 4bpw (rev 078455ff) + incoai DFlash2 (rev bf582e4e)
- Draft model (DFlash2)
- huggingface.co/incoai/GLM-5.3-Flash-DFlash2
- Engine
- TensorFold v0.6.0 (Mia's Python engine), tensor parallel on 2 GB10 boxes, EXL3 4 bpw with DFlash2 drafts — the program that loads the model and answers requests · github.com/ashhart/TensorFold
- Recipe and EXL3 quant
- github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold by Mia (MiaAI-Lab)
Steps
Get the recipe, the engine and the model files from the list above.
Start the server with this command in the recipe folder. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888.
PARALLEL=8 \ SERVED_NAME=GLM-5.3-Flash-EXL3 \ PORT=8888 \ HF_HUB_OFFLINE=1 ./start.shWe used the recipe at commit 3fd60dfc. Our run started it on the first DGX Spark (head); the second DGX Spark ran the worker. How the recipe starts the two DGX Sparks: see the README of the recipe, github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold.
What the parts mean
PORT=8888the network port the server listens on
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the start text exactly as our run recorded it, with the engine build in front, so anyone can repeat the result.
Mia Python engine TensorFold v0.6.0 + 107 patches (NOT the Zig engine), TP=2, DFlash2 drafts, in pin image tensorfold-glm53:pin-e66c657a433d (ghcr digest sha256:14b5072e...); PARALLEL=8 SERVED_NAME=GLM-5.3-Flash-EXL3 PORT=8888 PREPARE=0 PULL=0 HF_HUB_OFFLINE=1 ./start.sh ($HOME/glm-mia-recipe @3fd60dfc; scripts/local.sh: WORKER box-4, FABRIC_PEER, WORKER_WEIGHTS=nfs; head box-3 rank 0, worker box-4 rank 1; via spark-2x-glm-py3fd.sh start)Context and sampling: 1,048,576 tokens (the recipe default with an FP8 KV cache; the server log shows a window of 1,048,576 tokens), 8 requests at once (the recipe's default on two GB10 boxes is 4, it documents up to 8), vision on, DFlash2 drafts, seed 42. Sampling as in the recipe and the model's generation config, the same with thinking on and off: temperature 1.0, top_p 0.95. Speed, the quality tests and sparkDash with thinking off (GPQA 30 with thinking on); the build test with thinking on. The recipe's server default of 32,768 tokens per answer applies only when a request sets none; the build test set 65,536 (named on this page). Agent for the build test: OpenCode 1.19.3. Weights: the first GB10 box reads its copy, the second GB10 box reads the same files read-only over NFS (the recipe's own worker-weights mode). Licence note: the DFlash2 draft model is CC BY-NC-ND 4.0 (non-commercial) according to the recipe.. Hardware: 2x NVIDIA GB10 boxes linked by CX7 RDMA (about 109 Gb/s): 1x Gigabyte AI TOP ATOM and 1x Lenovo ThinkStation PGX. Both run driver 580.178.04 and kernel 7.0.0-1019-nvidia, GPU clock capped at 2200 MHz.. All raw files of this setup: raw/v2/glm53-flash-2sparks-mia-python.
Mia's own benchmark, run unchanged
This is Mia's own benchmark tool (commit 21a3511), run unchanged. It has its own section on this page, and it is never mixed with my numbers. A cut-off answer counts as wrong here, so a value can differ from sparkDash's own screen.
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| Quality QA 150 questions | 83.3 % ± 6.0 pts | 125 / 150 | 0 | summary | command |
| Quality MMLU 285 questions | 87.0 % ± 3.9 pts | 248 / 285 | 1 | summary | command |
| Tool Eval Quick 15 scenarios, tool use | 97 / 100 | 14 / 15 passed | — | tool eval result · tool eval report · tool eval command · summary | command |
| Decode, 1 request at once first token 143 ms | 110.8 tok/s | — | — | decode result · summary | command |
| Prefill at 4K first token 2.2 s | 2105.1 tok/s | — | — | prefill result · summary | command |
| Prefill at 8K first token 4.2 s | 2075.4 tok/s | — | — | prefill result · summary | command |
| Prefill at 16K first token 8.2 s | 2042.4 tok/s | — | — | prefill result · summary | command |
| Prefill at 32K first token 16.2 s | 2055.7 tok/s | — | — | prefill result · summary | command |
Seed 1234 applies to the two quality tests; decode, prefill and tool eval send no seed. The benchmark client ran on a separate machine that is never part of the measured group. Measured 2026-10-23. Version 2.50.0, commit 21a3511.
QA. Quality QA, 150 items, seed 1234, temperature 0, thinking off, 4 requests at once (Mia's default). An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
MMLU. Quality MMLU, 285 items (5 for each of 57 subjects, 4 answers), seed 1234, temperature 0, thinking off. An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
Decode. Decode bench, 1 request at once, 400 tokens, prompt type structured, temperature 0, thinking off. No seed is sent.
Prefill. Prefill bench at 4K, 8K, 16K and 32K tokens. sparkDash skips a size that does not fit into the context. No seed is sent.
Tool Eval Quick. Tool Eval Quick: tool-eval-bench (SeraphimSerapis, commit c61319b), 15 core scenarios (--short), --no-think, --timeout 120. Score 0 to 100. No seed is sent.
Note from the run: sparkDash runs its own tests (its own QA and MMLU questions at temperature 0, thinking off, a cut-off answer counts as wrong; none of the QA answers and 1 of the MMLU answers were cut off), so its decode and prefill values are not comparable with Speed v2 above: its decode test uses a structured prompt, Speed v2 a code and a prose prompt. The sparkDash client ran on another machine and was never part of the measured group.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 tools/quality2/sparkdash_client.py tools/quality2/cards/glm53-flash-exl3-4bpw-python-3fd60dfc-2x.json <box> --client <client machine> --job glm-py-2310 --out $RUNtool-eval-bench --base-url http://127.0.0.1:<port> --model GLM-5.3-Flash-EXL3 --short --no-think --timeout 120 --json-file $BENCH/sparkdash-client/runs/glm-py-2310-sparkdash-r6/tooleval/result.json --no-live