Home / 128 GB

128 GB · 1× DGX Spark (GB10)

0xWhiteMage's ThinkingCap Kearuga (Qwen3.8-27B, EXL3 4.00 bpw) in TandemLLM

1x NVIDIA DGX Spark

Recipe and EXL3 quant (Kearuga 4.00 bpw)0xWhiteMageX @0xWhiteMageGitHubHugging FaceWebsite
TandemLLM 0.12.2 (0xBakeer/TandemLLM, StairCut, community pack community/thinkingcap-27b-kearuga-exl3-4.00)commit TandemLLM v0.12.2 a84fdf211 GPUmeasured 2026-10-22independent check passed
Rank in 128 GB
5of 5
Overall score
67.4/ 100
Speed
22.3/ 35
Quality
23.1/ 25
Browser check
4.5/ 5
My rating
17.5/ 35

Overall score as of 22 Oct 2026. Scores are relative to the best run in the class today; a better model later moves older runs down.

How the overall score is calculated
PartThis runBest in classRelativeWeightPoints
Speed, one requestdecode 45.5 tok/sbest 48.3 tok/s94.3 %17.5 %16.5
Speed, 8 at oncetotal 41.5 tok/sbest 125.2 tok/s33.2 %17.5 %5.8
Qualitymean of 7 tests: 65.5 %best mean 70.8 %92.5 %25 %23.1
Browser check9 of 1090.0 %5 %4.5
My rating5 of 1050.0 %35 %17.5
Overall100 %67.4

The seven quality tests: MMLU-Pro 54.0 · GPQA 41.4 · IFEval 86.5 · LiveCodeBench 62.5 · long text 8K 75.0 · long text 32K 62.5 · GPQA 30 with thinking 76.7.

The speed part is half the decode speed with one request and half the total speed with 8 requests at once, each relative to the best run of this class; a run without a total for 8 requests at once gets the better of its decode speed alone and the mix with its highest measured total. Quality is relative to the best run of this class, the browser check and my rating are out of 10. The parts are rounded one by one. sparkDash, the memory test and speculative-decoding numbers are not part of it. How the score is made, and the assumption behind it.

Build test

What it builds as an agent

Task: Voxel world (3D scene, one HTML file). One attempt, no help. Counted: the final file counts.

The exact promptcopy it and try it yourself↓

This is the task prompt word for word, as we handed it to the agent harness. The harness adds its own system prompt and tools, and some harnesses add short context notes of their own. A run that goes past 90 minutes also gets a short “finish now” message, as the prompt says. To try it yourself, give it to your own coding agent: it needs a shell and a working folder.

Prompt: Voxel world (3D scene, one HTML file)
Design and create the most impressive, creative, elaborate and detailed
voxel art world you are capable of. The theme and style are ENTIRELY your
choice — pick whatever world you can execute best and push it as far as
you can: scale, variety, detail, color, animation, atmosphere, life.
Surprise me.

Hard requirements:
- Everything ends up in ONE single self-contained HTML file (game.html in
  the current working directory) that opens and runs directly in Chrome.
  Use WebGL and whatever libraries you like, but embed everything —
  no internet access at view time, no external files.
- The scene must present itself: include an automatic slow cinematic
  camera tour (orbiting the world, occasionally zooming in on highlights)
  that starts on load, plus mouse controls (orbit/zoom) for manual viewing.
- All user-facing text (UI, HUD, titles, descriptions, hints) must be in
  ENGLISH, regardless of the language of any project files you encounter.

When finished, verify your HTML actually loads and renders without
console errors, then state DONE with the absolute path of game.html.
Do not ask questions.

How you work (same for every model):
- You work alone in the folder /work (this is the current working directory). Nobody will answer questions. Aim to finish in about 60 minutes. After 90 minutes you will get a message to finish; then you have at most 30 more minutes. Hard stop after 120 minutes.
- Tools on this machine: shell, files, python3 with numpy and Pillow, ffmpeg, node, and `shot.sh game.html` (opens the page in a headless browser, saves a screenshot and prints console errors). There is no internet and no access to other machines.
- Keep a working /work/game.html at all times. If you are stopped, the last version that loaded without errors counts.
- JavaScript libraries available offline in the test browser, through an import map with these exact URLs: "three": "https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.module.js", "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/", "simplex-noise": "https://cdn.jsdelivr.net/npm/simplex-noise@4.0.3/dist/esm/simplex-noise.js", "@tweenjs/tween.js": "https://cdn.jsdelivr.net/npm/@tweenjs/tween.js@25.0.0/dist/tween.esm.js", "cannon-es": "https://cdn.jsdelivr.net/npm/cannon-es@0.20.0/dist/cannon-es.js". No other external resources.
- After the scene is built, set `window.scene` to the Three.js scene (if you use Three.js) and `window.ready = true`.
Browser check
9/ 10
My rating
5/ 10
Time
78.1min
Tokens
128,248
Harness
OpenCode 1.19.3
Screen recording of the page this run built, made in Chrome after the run. The page files are unchanged. Still image

Play it yourself

Open full screen

AI-generated The page this setup built, running live in your browser: click into it and use the controls it shows itself (usually drag to look around, scroll to zoom). It needs WebGL and can be slow on a weak graphics card.

The browser check

  • page loads without page error1 / 1
  • window.ready === true within 30 s1 / 1
  • no console errors (whole session)1 / 1
  • frame is not blank1 / 1
  • frame rate in software GL (rAF over 5 s)0 / 1
  • world: scene size 0.5 (parts >= 80 or >= 20000 triangles; canvas fallback), visible text English if any 0.51 / 1
  • automatic camera tour: the view changes on its own in two successive ~5 s windows (1 each)2 / 2
  • animation in the world (nodes / instances / particles moving besides the camera)1 / 1
  • OrbitControls: drag and wheel change the picture beyond the animation baseline1 / 1

Measured 2026-10-22. Raw files: check · run info · command · final choice · contract · sampling · gate stats · gate log · screenshot · clip.mp4 · web.mp4 · web-poster.jpg.

This is the rebuild of the showcase (22 Oct 2026). The first run of this setup (21 Oct) used a launcher that set OpenCode's answer limit to 253,952 tokens without an input limit. OpenCode 1.19.3 then summarised the conversation as soon as it passed 8,192 tokens (262,144 minus 253,952), and in that first run it did so 26 times. The rebuild uses the corrected launcher (opencode-limit-input), which also gives OpenCode the real input limit of 262,144 tokens, so it summarises only at 242,144 tokens. Model, engine, recipe, server settings and prompt are the same as in the first run. The first run's world is no longer shown here. Browser check 9.0 out of 10: the page loads, runs without console errors, is not blank, builds a world of 17,269 parts and 207,864 triangles with English text, animates, the automatic camera tour works (the picture changed on its own in both 5-second windows) and orbit control (drag and wheel) works. The one lost point is the frame rate: 5.4 frames per second in software rendering without a GPU, 20 are needed. The agent ended by itself after 78.1 minutes; the “finish now” message (at 90 minutes) was not needed. One try, no help. 19 versions of the page were seen, 14 loaded without errors, and the final version counts. The limit per answer was 253,952 tokens, the highest value the model card names (BottleCap card, README line 92). TandemLLM's default of 32,768 tokens applies only when a request sets none, so I raised the limit; the server setup is unchanged. My test gate shortened the limit of a request 67 times to what was left of the context. The run made 68 requests: 67 were the agent's own steps and 1 wrote the title. OpenCode did not summarise the conversation once (0 times, against 26 in the first run). The longest prompt of the run had 170,745 tokens; the model wrote 128,248 tokens in total. Thinking was on with the card's thinking sampling (temperature 1.0, top_p 0.95, top_k 20) and the server default effort xhigh.

Exact command we measured withfor reproducing↓

You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.

Build test (run show-r0)
run_showcase.sh --harness opencode-limit-input --task VX --upstream http://127.0.0.1:41797/v1 --model TandemLLM --sampling-file $RUN/sampling-show.json --out $RUN/raw/tc-neubau-l2-2210-show-r0/show --limit-s 7200 --inject-after-s 5400 --poll-s 20 --ctx 262144
Speed v2

How fast it runs

Decode
45.5tok/s
Decode, code
45.7tok/s
Decode, prose
45.4tok/s
500 words
≈ 15.6s
Prefill at 8K
829.1tok/s
First token at 2K
3.0s
MeasurementResult
Decode45.5 tok/s
Words per second32.0 words/s
Characters per second213.5 chars/s
Decode, code prompt45.7 tok/s
Decode, prose prompt45.4 tok/s
Spread of the decode runs (CV)9.6 %
Spread of the code runs (CV)7.3 %
Spread of the prose runs (CV)9.6 %
Decode, window 147.1 tok/s
Prefill at 2K671.8 tok/s
Prefill at 8K829.1 tok/s
Prefill at 32K790.0 tok/s
Time to first token at 2K3.0 s
Time to first token at 8K9.9 s
Time to first token at 32K41.5 s
Draft acceptance rate44.3 %
1 request at once (total)39.2 tok/s
2 at once (total)50.6 tok/s
4 at once (total)47.0 tok/s
8 at once (total)41.5 tok/s
1 request at once (per request)39.2 tok/s
2 at once (per request)34.2 tok/s
4 at once (per request)18.0 tok/s
8 at once (per request)9.8 tok/s
Server slots (from the recipe)1 slots

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once (later supplementary run of 2026-10-22, same setup): C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-10-21 (requests at once: 2026-10-22). Raw files: speed result · speed contract · log · speed result · speed contract · log.

Requests at once: the server has 1 slot (from the author's recipe). C2, C4, C8 exceed the slots: the extra requests wait in the queue, and they were measured anyway. The speed test ran with thinking off, the model card's sampling (temperature 1.0, top_p 0.95, top_k 20; the card names one set for thinking on and off) and the server default effort xhigh. The prompt for code ran at 45.65 tok/s, the prose prompt at 45.37 tok/s (medians of two runs each); the headline 45.51 tok/s is the mean of the two. The recipe runs one slot, so with 2, 4 and 8 requests at once the extra requests wait in the queue. At 2 requests at once the two rounds measured 63.4 and 37.8 tok/s in total (spread, CV, 35.7 %), and 40.4 and 27.9 tok/s per request (CV 25.9 %), so the 50.6 tok/s in total and the 34.2 tok/s per request for 2 at once are less certain than the other values.

Exact command we measured withfor reproducing↓

You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.

Speed v2 (run speed-r0)
python3 speed2.py --base-url http://127.0.0.1:40835/v1 --label run-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/plan-2110-speed-r0/work/speed2 --contract $RUN/raw/plan-2110-speed-r0/work/speed2/speed-contract.json --parts decode,prefill --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --tokenizer-json $HOME/thinkingcap-2110/target/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0} --sampling-source https://huggingface.co/0xWhiteMage/ThinkingCap-Qwen3.8-27B-Kearuga-EXL3-4.00bpw/blob/11447639033dd80a4c04536b5b37ed7d642802ee/README.md lines 178-181 (curl example: temperature 1.0, top_p 0.95, top_k 20, chat_template_kwargs enable_thinking true + reasoning_effort xhigh) and generation_config.json lines 9-11 (temperature 1.0, top_k 20, top_p 0.95); min_p 0.0 from https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-GGUF README.md line 100 (rev dc3aa8563d5d48d1d83637eb67147e1b52215cf7). The card gives no separate non-thinking set: the same values are used with thinking off. presence_penalty is not named by any ThinkingCap card -> not sent. --model TandemLLM --no-cache-prompt --server-log $RUN/logs/server-A.log --engine-version-file $RUN/logs/engine-version.txt

# Requests at once: supplementary run of 2026-10-22 (raw folder speed-r0-conc)
python3 speed2.py --base-url http://127.0.0.1:37237/v1 --label run-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/plan-2210-speed-r0/work/speed2 --contract $RUN/raw/plan-2210-speed-r0/work/speed2/speed-contract.json --parts conc --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --tokenizer-json $HOME/thinkingcap-2110/target/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0} --sampling-source https://huggingface.co/0xWhiteMage/ThinkingCap-Qwen3.8-27B-Kearuga-EXL3-4.00bpw/blob/11447639033dd80a4c04536b5b37ed7d642802ee/README.md lines 178-181 (curl example: temperature 1.0, top_p 0.95, top_k 20, chat_template_kwargs enable_thinking true + reasoning_effort xhigh) and generation_config.json lines 9-11 (temperature 1.0, top_k 20, top_p 0.95); min_p 0.0 from https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B-GGUF README.md line 100 (rev dc3aa8563d5d48d1d83637eb67147e1b52215cf7). The card gives no separate non-thinking set: the same values are used with thinking off. presence_penalty is not named by any ThinkingCap card -> not sent. --model TandemLLM --no-cache-prompt --server-log $RUN/logs/server-A.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 1 --server-slots-source 0xBakeer/TandemLLM@a84fdf21 (v0.12.2) ops/serve.env:322 (QSE_PARALLEL=1) --supplement-of run-speed-r0
Quality v2

How well it answers

TestResult95 % marginRightCut offcount as wrongRaw files
MMLU-Pro
400 fixed questions, letter only
54.0 % ± 4.9 pts216 / 4000answers · summary · log · contractcommand
GPQA Diamond
198 questions, letter only
41.4 % ± 6.8 pts82 / 1980answers · summary · log · contractcommand
IFEval
200 fixed prompts, strict
86.5 % ± 4.7 pts173 / 2000answers · summary · log · contractcommand
LiveCodeBench
40 newest tasks, the code runs
62.5 % ± 14.4 pts25 / 402summary · log · contractcommand
Long text 8K
4 tasks x 10 at 8192 tokens
75.0 % ± 13.0 pts30 / 400log · contract · answers · summarycommand
Long text 32K
4 tasks x 10 at 32768 tokens
62.5 % ± 14.4 pts25 / 400log · contract · answers · summarycommand
GPQA 30, thinking on
30 questions, limit 16,384 tokens
76.7 % ± 14.6 pts
median 2,105 tokens, 41.6 s per question
23 / 305log · contract · answers · summarycommand

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-10-21.

Note from the run: 5 of 30 GPQA answers with thinking were cut off at their limit of 16,384 tokens and 2 of 40 LiveCodeBench answers at their limit of 32,768 tokens; these count as wrong. In the other quality-v2 tests (MMLU-Pro, GPQA Diamond (letter only), IFEval and the long-text tests) no answer was cut off. In Mia's sparkDash MMLU part 5 of 285 answers were cut off at its 1,024-token limit and count as wrong too. GPQA Diamond ran on the public mirror fingertap/GPQA-Diamond (the original is gated). These are my own tests of this setup.

Exact commands we measured withfor reproducing↓

You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.

MMLU-Pro (run inst-r1)
bash tools/quality2/measure_all.sh tools/quality2/cards/thinkingcap-27b-kearuga-exl3-4.0bpw-tandemllm0122-1x.json --parts mmlupro --job plan-2110
GPQA Diamond (run inst-r1)
bash tools/quality2/measure_all.sh tools/quality2/cards/thinkingcap-27b-kearuga-exl3-4.0bpw-tandemllm0122-1x.json --parts gpqa --job plan-2110
IFEval (run inst-r1)
bash tools/quality2/measure_all.sh tools/quality2/cards/thinkingcap-27b-kearuga-exl3-4.0bpw-tandemllm0122-1x.json --parts ifeval --job plan-2110
LiveCodeBench (run inst-r1)
bash tools/quality2/measure_all.sh tools/quality2/cards/thinkingcap-27b-kearuga-exl3-4.0bpw-tandemllm0122-1x.json --parts lcb --job plan-2110
Long text 8K (run inst-r1)
bash tools/quality2/measure_all.sh tools/quality2/cards/thinkingcap-27b-kearuga-exl3-4.0bpw-tandemllm0122-1x.json --parts ruler --job plan-2110
Long text 32K (run inst-r1)
bash tools/quality2/measure_all.sh tools/quality2/cards/thinkingcap-27b-kearuga-exl3-4.0bpw-tandemllm0122-1x.json --parts ruler --job plan-2110
GPQA 30, thinking on (run think-r2)
bash tools/quality2/measure_all.sh tools/quality2/cards/thinkingcap-27b-kearuga-exl3-4.0bpw-tandemllm0122-1x.json --parts think --job plan-2110
Memory test

How much context fits

Largest size tested
262,144tokens
Peak memory
68.22GiB
Decode at that size
21.9tok/s
Context sizePrompttokensFilledFirst tokenPrefilltok/s, as the server reports itDecodetok/sPeak memoryGiBResult
4K 4,096 tokens3,89795.1 %4.7 s82533.438.14answered (stop)
8K 8,192 tokens7,78895.1 %9.2 s84929.539.63answered (stop)
16K 16,384 tokens15,57095.0 %10.6 s79227.440.92answered (stop)
32K 32,768 tokens31,13595.0 %20.2 s78127.943.46answered (stop)
64K 65,536 tokens62,26995.0 %44.0 s71930.447.58answered (stop)
128K 131,072 tokens124,54595.0 %102.6 s61722.954.93answered (stop)
256K 262,144 tokens249,08595.0 %267.7 s46821.968.22answered (stop)

Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed is measured on that answer at the largest size. Peak memory is the rise of used memory (MemTotal - MemAvailable) over the idle value before the step, on the one box. Unified memory, so GPU and CPU share it. Measured 2026-10-21. Raw files: result of every step · log · contract.

Note from the run: The largest size tested is 262,144 tokens, the setup's own context, so this is the largest size tested, not necessarily the largest the engine could load. All seven sizes loaded and answered; decode at the seven sizes ranges from 21.89 to 33.38 tok/s (21.89 at 262,144). The memory number is the rise over the idle value (unified memory) at the largest step and includes the model weights.

Exact command we measured withfor reproducing↓

You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.

Memory test (run fit-r3)
bash tools/quality2/measure_all.sh tools/quality2/cards/thinkingcap-27b-kearuga-exl3-4.0bpw-tandemllm0122-1x.json --parts fit --job plan-2110
Credits

Also made this possible

Setup

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware
1x NVIDIA DGX Spark
Model
0xWhiteMage/ThinkingCap-Qwen3.8-27B-Kearuga-EXL3-4.00bpw @11447639 (weights byte-equal to the recipe pin d6df32fa) (ThinkingCap by BottleCap AI on Qwen3.8-27B)
ThinkingCap fine-tune
huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B
Engine
TandemLLM 0.12.2 on one DGX Spark, reasoning effort xhigh — the program that loads the model and answers requests · github.com/0xBakeer/TandemLLMBuild and settings we used: TandemLLM 0.12.2 (0xBakeer/TandemLLM, StairCut, community pack community/thinkingcap-27b-kearuga-exl3-4.00)
Recipe and EXL3 quant (Kearuga 4.00 bpw)
huggingface.co/0xWhiteMage/ThinkingCap-Qwen3.8-27B-Kearuga-EXL3-4.00bpw by 0xWhiteMage
Context window
262,144 tokens: how much text the model can see at once

Steps

  1. Get the recipe, the engine and the model files from the list above.

  2. Start the server with this command. It loads the model into memory.

    ops/start.sh with ops/active-profile = community/thinkingcap-27b-kearuga-exl3-4.00 (ops/use-profile.sh community/thinkingcap-27b-kearuga-exl3-4.00), run/local.env \
      REASONING_EFFORT=xhigh; engine :8000; requests via :8888 (adds X-Tandem: raw); via GB10 box 1-thinkingcap-tandemllm-0122.sh start
Exact command we measured withfor reproducing↓

You do not need this to run the model. It is the start text exactly as our run recorded it, with the engine build in front, so anyone can repeat the result.

Start, as recorded with the run
TandemLLM 0.12.2 (0xBakeer/TandemLLM, StairCut, community pack community/thinkingcap-27b-kearuga-exl3-4.00); ops/start.sh with ops/active-profile = community/thinkingcap-27b-kearuga-exl3-4.00 (ops/use-profile.sh community/thinkingcap-27b-kearuga-exl3-4.00), run/local.env REASONING_EFFORT=xhigh; engine :8000; requests via :8888 (adds X-Tandem: raw); via GB10 box 1-thinkingcap-tandemllm-0122.sh start

Context and sampling: 262144 (ops/serve.env MAX_LEN), 1 slot (QSE_PARALLEL=1), TandemLLM drafters (K3 EXL3 + K4 draft head + BF16 DFlash2 b8/b16), seed 42, model-card sampling (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0; presence_penalty not named, not sent), thinking off in the speed and instruct parts, thinking on in think and show. Reasoning effort xhigh: TandemLLM ships medium, so the server default was set to xhigh (REASONING_EFFORT=xhigh in run/local.env, the only change to the community profile); the model's chat template defaults to xhigh too (chat_template.jinja:47) and the model cards recommend it. Our measuring code sends no reasoning-effort field and OpenCode sends none either, so the server default applies; we checked that the default and an explicit xhigh give the same prompt. Requests went through a small pass-through proxy on port 8888 that only adds the header X-Tandem: raw, because the engine's benchmark docs say a benchmark runs the plain engine (docs/benchmarks.md:15, docs/server.md:244). While setting up, the first activation of the community profile (ops/use-profile.sh) rolled itself back once after an empty smoke answer; the cause was not found, the start through our wrapper script then ran cleanly and the run used the profile as installed. No prompt-cache help (a nonce at the prompt start; the engine's own prefix/session cache is on by default). Truncated answers count as wrong. Hardware: 1x NVIDIA DGX Spark. All raw files of this setup: raw/v2/thinkingcap-27b-exl3-1spark-tandemllm.

Mia's sparkDash 2.50.0

Mia's own benchmark, run unchanged

This is Mia's own benchmark tool (commit 21a3511), run unchanged. It has its own section on this page, and it is never mixed with my numbers. A cut-off answer counts as wrong here, so a value can differ from sparkDash's own screen.

Tool Eval Quick
97/ 100
Decode, 1 request
161.7tok/s
TestResult95 % marginRightCut offcount as wrongRaw files
Quality QA
150 questions
78.7 % ± 6.5 pts118 / 1500answers · summarycommand
Quality MMLU
285 questions
88.1 % ± 3.8 pts251 / 285
sparkDash's own screen counts 252 right
5answers · summarycommand
Tool Eval Quick
15 scenarios, tool use
97 / 10014 / 15 passed—tool eval result · tool eval report · tool eval command · summarycommand
Decode, 1 request at once
first token 137 ms
161.7 tok/s——decode result · summarycommand
Prefill at 4K
first token 5.2 s
790.6 tok/s——prefill result · summarycommand
Prefill at 8K
first token 10.0 s
820.1 tok/s——prefill result · summarycommand
Prefill at 16K
first token 20.0 s
820.8 tok/s——prefill result · summarycommand
Prefill at 32K
first token 41.1 s
799.8 tok/s——prefill result · summarycommand

Seed 1234 applies to the two quality tests; decode, prefill and tool eval send no seed. The benchmark client ran on a separate machine that is never part of the measured group. Measured 2026-10-21. Version 2.50.0, commit 21a3511.

QA. Quality QA, 150 items, seed 1234, temperature 0, thinking off, 4 requests at once (Mia's default). An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).

MMLU. Quality MMLU, 285 items (5 for each of 57 subjects, 4 answers), seed 1234, temperature 0, thinking off. An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).

Decode. Decode bench, 1 request at once, 400 tokens, prompt type structured, temperature 0, thinking off. No seed is sent.

Prefill. Prefill bench at 4K, 8K, 16K and 32K tokens. sparkDash skips a size that does not fit into the context. No seed is sent.

Tool Eval Quick. Tool Eval Quick: tool-eval-bench (SeraphimSerapis, commit c61319b), 15 core scenarios (--short), --no-think, --timeout 120. Score 0 to 100. No seed is sent.

Note from the run: sparkDash 2.50.0 (@21a3511) on a client machine (never in the measured group), Quality: seed 1234, temperature 0, thinking off, a truncated answer counts as wrong; decode, prefill and tool eval send no seed; own row, never mixed with quality-v2.

Exact commands we measured withfor reproducing↓

You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.

sparkDash (run sparkdash-r5)
python3 tools/quality2/sparkdash_client.py tools/quality2/cards/thinkingcap-27b-kearuga-exl3-4.0bpw-tandemllm0122-1x.json <box> --client <client machine> --job plan-2110 --out $RUN
Tool Eval Quick, as sparkDash starts it
tool-eval-bench --base-url http://127.0.0.1:<port> --model TandemLLM --short --no-think --timeout 120 --json-file $BENCH/sparkdash-client/runs/work/tooleval/result.json --no-live