Inkling-2-Small (594B) on four DGX Sparks: 0xSero's engine, own build for the GB10

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of about 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; the local tokenizer did not start, so the lengths were estimated, not counted: the steps held 17.5K, 35.1K, 70.3K and 143.6K prompt tokens (median, as counted by the server), the prefill steps 2.2K, 8.9K and 35.5K; decode = tokens after the first / time after the first token; prefill = prompt tokens actually sent / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-15. Raw files: speed result · speed contract · log.

What ran. 0xSero's Inkling engine (inkling2-exl3) at commit 5cf1465d with expert parallelism 2 x tensor parallelism 2 for the experts and tensor parallelism 4 for attention, b12x fused MoE kernels, an FP8 draft and speculative decoding that is always on, serving Inkling-2-Small (594B parameters, 17B active; Thinking Machines Lab) in the EXL3 4.0 bpw files of 0xSero (revision e58fa769), on four DGX Sparks in a ring without a switch. Where this run differs from the author's setup. (1) The author's package targets four RTX PRO 6000 cards (x86_64, CUDA 12.8, sm_120); there is no image for the GB10. We built exllamav3 v1.6.6 plus the author's overlay for sm_121a inside NVIDIA's NGC image and applied two patches of ours (public in our recipe): one cuts and stores the expert slices each GB10 box needs (each box downloads 73.7 GB of expert byte ranges instead of the whole 312 GB checkpoint) and closes the safetensors handles after every layer, because otherwise the boxes run out of memory while loading; the other maps the local rank to one GPU per host, and maps enable_thinking false to reasoning effort none so that our test tools can talk to the server. (2) NCCL_CUMEM_ENABLE is 1 instead of 0, otherwise NCCL does not connect on the switchless ring. (3) The KV pool is fixed at 24 GiB per rank (--kv-gb 24), because the GB10 has unified memory. (4) Token-sharded prefill and peer all-reduce are switched off. (5) The speed test uses a window of 380,000 tokens per request, the server default, so every context step up to 128K fits. The model writes its reasoning first even at reasoning effort none, so the 512 tokens of the speed test hold only reasoning text: the page has no words-per-second value and no '500 words' time. Against the author's numbers. The author's README (version at the time of our run) lists, for four RTX PRO 6000 cards and his own served sweep with 16 prompts, 220 / 328 / 421 / 540 / 697 tok/s in total with 1 / 2 / 4 / 8 / 16 streams. With his sweep script on our four GB10 boxes at commit 5cf1465d we measured 52.6 / 78.9 / 103.7 / 131.6 / 167.4 tok/s (about a quarter of his numbers at every level); the engine accepts 2.84 tokens per step with one stream. His 16 prompts are not in his repository, so we wrote 16 chat, code and agent prompts with up to 2,048 new tokens. The sweep is not part of the published raw files of this page; the speed numbers on the page come from our own speed test.

python3 speed2.py --base-url http://127.0.0.1:44185/v1 --label inkling4-server-v4-1511-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/inkling4-server-v4-1511-speed-r0/inkling4-server-v4-1511-speed-r0/speed2 --contract $RUN/inkling4-server-v4-1511-speed-r0/inkling4-server-v4-1511-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,ctx --prefill-depths 2048,8192,32768,65536,131072 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/inkling4-1411/v3/model/tokenizer.json --sampling {"temperature": 0.0, "top_p": 1.0} --sampling-source Author's headline numbers are temperature 0 (README); the author server honours temperature. thinkingmachines/Inkling-2-Small names no sampling. Thinking off = reasoning_effort none (patch 0004 maps enable_thinking false). --model inkling-2-small --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 16 --server-slots-source author's server i2_server.py --slots 16 (default); author's sweep concurrencies 1,2,4,8,16: bench/i2_sweep.py --server-ctx 131072 --server-ctx-source card GB10 box 3-inkling2-small-exl3-4.0bpw-ep2tp2-b12x-4x-server-v4-1511 field ctx

How well it answers

Quality state. The speed numbers on this page were measured at the engine commit 5cf1465d. The quality numbers were measured earlier, at the previous engine state 8f12aa4e, and were not repeated at 5cf1465d (jobs inkling4-q2b-v3-1511 and inkling4-q2c-v3-1511). Reasoning effort none, fixed lists, seed 42. The model still writes thinking first, so MMLU-Pro and GPQA got an answer limit of 4,096 tokens, the same as the server's default, and LiveCodeBench a limit of 32,768 tokens. The commands under the quality cells name the setup card of the speed run; the quality jobs ran with the card of the earlier engine state (card name ending in server-v3-1511, same server settings except the engine commit and the adaptive verification). A cut-off answer counts as wrong: the table shows how many answers were cut. We did not measure the unquantised original, so we cannot say how much the quantisation costs.

Reasoning effort none (the model still writes thinking first). Seed 42. Temperature 0, top-p 1.0 (the model card names no sampling; the author's headline numbers use temperature 0). A cut-off answer counts as wrong. Measured 2026-11-15.

bash tools/quality2/measure_all.sh tools/quality2/cards/inkling2-small-exl3-4.0bpw-ep2tp2-b12x-4x-server-v4-1511.json --parts mmlupro --job inkling4-q2b-v3-1511

bash tools/quality2/measure_all.sh tools/quality2/cards/inkling2-small-exl3-4.0bpw-ep2tp2-b12x-4x-server-v4-1511.json --parts gpqa --job inkling4-q2b-v3-1511

bash tools/quality2/measure_all.sh tools/quality2/cards/inkling2-small-exl3-4.0bpw-ep2tp2-b12x-4x-server-v4-1511.json --parts ifeval --job inkling4-q2b-v3-1511

bash tools/quality2/measure_all.sh tools/quality2/cards/inkling2-small-exl3-4.0bpw-ep2tp2-b12x-4x-server-v4-1511.json --parts lcb --job inkling4-q2c-v3-1511

bash tools/quality2/measure_all.sh tools/quality2/cards/inkling2-small-exl3-4.0bpw-ep2tp2-b12x-4x-server-v4-1511.json --parts ruler --job inkling4-q2b-v3-1511

bash tools/quality2/measure_all.sh tools/quality2/cards/inkling2-small-exl3-4.0bpw-ep2tp2-b12x-4x-server-v4-1511.json --parts ruler --job inkling4-q2b-v3-1511

Also made this possible

• Model files (EXL3 4.0 bpw, revision e58fa769)00xSero@0xSero Hugging Face
• Model (Inkling 2 Small)TThinking Machines Lab@thinkymachines Hugging Face
• Quantisation format (EXL3, exllamav3 v1.6.6)turboderp@turboderp_ GitHub Hugging Face
• Our public recipe for the four GB10 boxes (patches, scripts, steps)NNipale-aiGitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

4x NVIDIA DGX Spark (GB10, 128 GB unified memory each, 512 GB in total), switchless CX7 ring with 2 links between each neighbouring pair (8 links, each measured at 109 Gb/s), driver 580.178.04, kernel 7.0.0-1019 (nvidia)

Model

Thinking Machines Inkling-2-Small (594B total, 17B active) in 0xSero/Inkling-2-Small-EXL3-4bpw @e58fa769cd946a7df2e2bbd5bec790290549b150

Model files (EXL3 4.0 bpw, revision e58fa769)

huggingface.co/0xSero/Inkling-2-Small-EXL3-4bpw

Engine

0xSero's inkling2-exl3 at commit 5cf1465d with our two patches, built for the GB10 — the program that loads the model and answers requests · github.com/0xSero/inkling2-exl3

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory.

   Copy

   PC_SHARD=0 \
     PEER_AR=0 bash start_server.sh

0xSero/inkling2-exl3 @5cf1465d (overlay on exllamav3 v1.6.6: b12x fused Trellis MoE, EP2xTP2, FP8 draft, adaptive verification, speculative decoding always on) + patches 0003/0004 + the author's OpenAI server; PC_SHARD=0 PEER_AR=0 bash start_server.sh (one container per GB10 box, rank r on the r-th GB10 box of the ring; the script and the patches are in the public recipe, github.com/Nipale-ai/inkling2-exl3-dgx-spark; engine 0xSero/inkling2-exl3 at 5cf1465d, exllamav3 v1.6.6, FlashInfer b12x kernels at 9c509967; token-sharded prefill and peer all-reduce are off)