niklaslenzbertholomus's TP2 recipe: DeepSeek-V4.1-Flash-1017
2x NVIDIA DGX Spark class boxes, tensor parallel 2 over two CX7 cables with RDMA: the first (rank 0) and the second (rank 1, an ASUS Ascent GX10), GPU clock capped at 2200 MHz. The ASUS GX10 runs an older driver (580.173.02) and kernel (6.17.0-1031-nvidia) than the first box (580.178.04, 7.0.0-1019-nvidia). That is named openly.
Overall score as of 24 Oct 2026. Scores are relative to the best run in the class today; a better model later moves older runs down.
How the overall score is calculated
| Part | This run | Best in class | Relative | Weight | Points |
|---|---|---|---|---|---|
| Speed, one request | decode 70.6 tok/s | best 70.6 tok/s | 100.0 % | 17.5 % | 17.5 |
| Speed, 8 at once | total 106.3 tok/s | best 106.3 tok/s | 100.0 % | 17.5 % | 17.5 |
| Quality | mean of 7 tests: 66.1 % | best mean 69.1 % | 95.7 % | 25 % | 23.9 |
| Browser check | 8 of 10 | 80.0 % | 5 % | 4.0 | |
| My rating | 5.5 of 10 | 55.0 % | 35 % | 19.3 | |
| Overall | 100 % | 82.2 |
The seven quality tests: MMLU-Pro 62.0 · GPQA 43.9 · IFEval 92.0 · LiveCodeBench 60.0 · long text 8K 67.5 · long text 32K 57.5 · GPQA 30 with thinking 80.0.
The speed part is half the decode speed with one request and half the total speed with 8 requests at once, each relative to the best run of this class; a run without a total for 8 requests at once gets the better of its decode speed alone and the mix with its highest measured total. Quality is relative to the best run of this class, the browser check and my rating are out of 10. The parts are rounded one by one. sparkDash, the memory test and speculative-decoding numbers are not part of it. How the score is made, and the assumption behind it.
What it builds as an agent
Task: Voxel world (3D scene, one HTML file). One attempt, no help. Counted: the final file counts.
The exact promptcopy it and try it yourself↓
This is the task prompt word for word, as we handed it to the agent harness. The harness adds its own system prompt and tools, and some harnesses add short context notes of their own. A run that goes past 90 minutes also gets a short “finish now” message, as the prompt says. To try it yourself, give it to your own coding agent: it needs a shell and a working folder.
Design and create the most impressive, creative, elaborate and detailed
voxel art world you are capable of. The theme and style are ENTIRELY your
choice — pick whatever world you can execute best and push it as far as
you can: scale, variety, detail, color, animation, atmosphere, life.
Surprise me.
Hard requirements:
- Everything ends up in ONE single self-contained HTML file (game.html in
the current working directory) that opens and runs directly in Chrome.
Use WebGL and whatever libraries you like, but embed everything —
no internet access at view time, no external files.
- The scene must present itself: include an automatic slow cinematic
camera tour (orbiting the world, occasionally zooming in on highlights)
that starts on load, plus mouse controls (orbit/zoom) for manual viewing.
- All user-facing text (UI, HUD, titles, descriptions, hints) must be in
ENGLISH, regardless of the language of any project files you encounter.
When finished, verify your HTML actually loads and renders without
console errors, then state DONE with the absolute path of game.html.
Do not ask questions.
How you work (same for every model):
- You work alone in the folder /work (this is the current working directory). Nobody will answer questions. Aim to finish in about 60 minutes. After 90 minutes you will get a message to finish; then you have at most 30 more minutes. Hard stop after 120 minutes.
- Tools on this machine: shell, files, python3 with numpy and Pillow, ffmpeg, node, and `shot.sh game.html` (opens the page in a headless browser, saves a screenshot and prints console errors). There is no internet and no access to other machines.
- Keep a working /work/game.html at all times. If you are stopped, the last version that loaded without errors counts.
- JavaScript libraries available offline in the test browser, through an import map with these exact URLs: "three": "https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.module.js", "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/", "simplex-noise": "https://cdn.jsdelivr.net/npm/simplex-noise@4.0.3/dist/esm/simplex-noise.js", "@tweenjs/tween.js": "https://cdn.jsdelivr.net/npm/@tweenjs/tween.js@25.0.0/dist/tween.esm.js", "cannon-es": "https://cdn.jsdelivr.net/npm/cannon-es@0.20.0/dist/cannon-es.js". No other external resources.
- After the scene is built, set `window.scene` to the Three.js scene (if you use Three.js) and `window.ready = true`.
Play it yourself
Open full screenAI-generated The page this setup built, running live in your browser: click into it and use the controls it shows itself (usually drag to look around, scroll to zoom). It needs WebGL and can be slow on a weak graphics card.
The browser check
- page loads without page error1 / 1
- window.ready === true within 30 s1 / 1
- no console errors (whole session)1 / 1
- frame is not blank1 / 1
- frame rate in software GL (rAF over 5 s)0 / 1
- world: scene size 0.5 (parts >= 80 or >= 20000 triangles; canvas fallback), visible text English if any 0.51 / 1
- automatic camera tour: the view changes on its own in two successive ~5 s windows (1 each)2 / 2
- animation in the world (nodes / instances / particles moving besides the camera)1 / 1
- OrbitControls: drag and wheel change the picture beyond the animation baseline0 / 1
Measured 2026-10-24. Raw files: check · run info · command · final choice · contract · sampling · gate stats · gate log · check · contract · screenshot · clip.mp4 · web.mp4 · web-poster.jpg.
Browser check 8.0 out of 10 (voxel-world task): the page loads without a page error, is ready after 0.01 s, has no console errors, is not blank, shows a world (173,527 parts, 2,085,288 triangles, English text), flies an automatic camera tour in both windows and animates (25 animated nodes). The 2 lost points are the frame rate (0.18 frames per second in software rendering without a GPU, 20 are needed for the full point) and the orbit-controls test (drag and wheel changed the picture less than the check needs). How this score came about: at the end of the run, the run's own check did not write a score, because its closing screenshots stalled in software rendering (the screenshot call timed out). On 24 Oct 2026 I re-ran the same check on the same page file (sha256 4450c30b...) with a changed capture method: the screenshot is taken through the browser's DevTools protocol (captureScreenshot with fromSurface set to false) instead of the usual screenshot call. The checks and their thresholds are the same. The score and the check details come from that re-run (folder check-recheck, with its own contract and check.json); every other number of the run stays from the run itself. The agent (DeepSeek Harness 0.2.1-rc.2, the harness DeepSeek ships) finished by itself after 73.7 minutes, one try, no help; the “finish now” message (from 90 minutes) was not needed. 24 versions of the page were seen on the way, all 24 loaded without errors, and the last one counts. 95 requests, no error, no request blocked (11,730,259 prompt tokens, 138,675 answer tokens); 93 requests ended with a tool call and 2 with a normal stop, none at the limit; the longest answer had 46,330 tokens, the last prompt had 190,984 tokens (gate-log.jsonl has every request). Some calls of the harness itself asked for 4,096 tokens; the test proxy lowered none of them. The limit per answer was 262,144 tokens, the value on the DeepSeek model card (README lines 77 to 84), and no request was lowered. The recipe's server default is 32,768 tokens and applies only when a request sets none, so I raised the limit for this test (the server setup is unchanged). Thinking was on with the recipe's sampling (temperature 1.0, top_p 0.95, top_k 20), the context was 1,048,576 tokens. The agent container ran on the second GB10 box, the test proxy on the first.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
run_showcase.sh --harness dsh-021 --task VX --upstream http://127.0.0.1:18080/v1 --model DeepSeek-V4.1-Flash-1017-TP2 --sampling-file $RUN/raw/bert-tp2-2410-show-r16/show.in/sampling-show.json --out $RUN/raw/bert-tp2-2410-show-r16/show --limit-s 7200 --inject-after-s 5400 --poll-s 20 --ctx 1048576How fast it runs
| Measurement | Result |
|---|---|
| Decode | 70.6 tok/s |
| Words per second | 49.8 words/s |
| Characters per second | 325.1 chars/s |
| Decode, code prompt | 70.5 tok/s |
| Decode, prose prompt | 70.6 tok/s |
| Spread of the decode runs (CV) | 6.5 % |
| Spread of the code runs (CV) | 6.5 % |
| Spread of the prose runs (CV) | 1.5 % |
| Decode, window 1 | 72.7 tok/s |
| Prefill at 2K | 1728.7 tok/s |
| Prefill at 8K | 2156.9 tok/s |
| Prefill at 32K | 1830.0 tok/s |
| Time to first token at 2K | 1.2 s |
| Time to first token at 8K | 3.8 s |
| Time to first token at 32K | 17.9 s |
| Draft acceptance rate | 57.3 % |
| Mean accepted length | 2.4 tokens |
| 1 request at once (total) | 61.6 tok/s |
| 2 at once (total) | 78.9 tok/s |
| 4 at once (total) | 107.4 tok/s |
| 8 at once (total) | 106.3 tok/s |
| 1 request at once (per request) | 61.6 tok/s |
| 2 at once (per request) | 40.3 tok/s |
| 4 at once (per request) | 27.7 tok/s |
| 8 at once (per request) | 18.0 tok/s |
| Server slots (from the recipe) | 4 slots |
| Long answer, wall time | 53.8 s |
Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-10-24. Raw files: speed result · speed contract · log · long answer.
Requests at once: the server has 4 slots (from the author's recipe). C8 exceeds the slots: the extra requests wait in the queue, and they were measured anyway. This run used the recipe at v0.7.2, commit 30a5bf9. The slot remark in speed2.json (and so in the command below, --server-slots-source) still quotes the text of the earlier pin, v0.7.0 (54eca2fd); the slot setting it cites, PARALLEL=4 (config.env.example:36), is the same 4 in the v0.7.2 recipe. The raw files are published as they were written. The speed test ran with the author's recipe (bertholomus/deepseek-v4.1-tensorfold-tp2-2xgb10, commit 30a5bf9, v0.7.2 of 24 Oct 2026, 05:51 UTC): TensorFold's native engine (branch deepseek-v41-zig-tp2, engine commit a57deb9d, built here from that branch with its test suite passing, the same binaries on both GB10 boxes), tensor parallel over two GB10 boxes (two CX7 cables, RDMA), DSpark drafts as the recipe sets them, a context window of 1,048,576 tokens and 4 requests at once (PARALLEL=4, config.env.example:36). The author documents no higher setting for the two-Spark lane, so 4 is the largest; the 8-requests-at-once level goes beyond the 4 slots, the extra requests wait in the queue, and it was measured anyway. Sampling is the one in the recipe: temperature 1.0, top_p 0.95, top_k 20; seed 42; thinking off per request; no prompt cache. The headline 70.55 tok/s is the mean of the code prompt (70.53) and the prose prompt (70.57), medians of two runs each (spread 6.5 % and 1.5 %). Network: the recipe opens GID index 5 of each RDMA device. On our boxes that index did not exist until the link ports had IPv4 addresses, so I gave all four link ports one (a /30 network per cable) before the measurements; it is not kept over a reboot. Nothing else on the network side was changed. One GB10 box (an ASUS GX10) runs an older driver and kernel than the other (named under hardware above).
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 speed2.py --base-url http://127.0.0.1:44495/v1 --label bert-tp2-2410-speed-r13 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/bert-tp2-2410-speed-r13/bert-tp2-2410-speed-r13/speed2 --contract $RUN/raw/bert-tp2-2410-speed-r13/bert-tp2-2410-speed-r13/speed2/speed-contract.json --parts decode,prefill,conc,l1 --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --tokenizer-json $HOME/bert-2210/model/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 1.0, "top_p": 0.95, "top_k": 20} --sampling-source The author's served values: bertholomus/deepseek-v4.1-tensorfold-tp2-2xgb10@30a5bf9 config.env.example:27 SERVER_FLAGS '--thinking --max-tokens 32768 --temperature 1.0 --top-p 0.95 --top-k 20' (a request's own values win; our config.env raises only the max-tokens default to 262144). The base card deepseek-ai/DeepSeek-V4.1-Flash-1017 README.md:77-84 recommends top_p 1.0; the recipe's served 0.95 wins (SENDEPLAN 8: run it as the author runs it today), same decision as the TP4 card. Speed v2 runs with thinking off (enable_thinking false per request), show/think with thinking on. --model DeepSeek-V4.1-Flash-1017-TP2 --no-cache-prompt --server-log $RUN/logs/server-A.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 4 --server-slots-source bertholomus/deepseek-v4.1-tensorfold-tp2-2xgb10@54eca2fd66548d9f83d68c04f5a5e2222af972e0 (v0.7.0, 2026-10-23 18:42 UTC, engine line deepseek-v41-zig-tp2) config.env.example:36 (PARALLEL=4, requests rank 0 serves at once), :34 (STREAMS=4), README.md:48 ('2,097,152 positions, 4 streams, --parallel 4', the served lane everywhere); the author documents no higher PARALLEL for the TP2 served lane, so 4 is both the author's default and the only named setting (SENDEPLAN 7.3: never 'not measured' because of unknown slots - the slots are known; C1/C2/C4/C8 are measured as always).How well it answers
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| MMLU-Pro 400 fixed questions, letter only | 62.0 % ± 4.7 pts | 248 / 400 | 49 | answers · summary · log · contract | command |
| GPQA Diamond 198 questions, letter only | 43.9 % ± 6.9 pts | 87 / 198 | 46 | answers · summary · log · contract | command |
| IFEval 200 fixed prompts, strict | 92.0 % ± 3.8 pts | 184 / 200 | 0 | answers · summary · log · contract | command |
| LiveCodeBench 40 newest tasks, the code runs | 60.0 % ± 14.5 pts | 24 / 40 | 2 | summary · log · contract | command |
| Long text 8K 4 tasks x 10 at 8192 tokens | 67.5 % ± 14.0 pts | 27 / 40 | 3 | log · contract · answers · summary | command |
| Long text 32K 4 tasks x 10 at 32768 tokens | 57.5 % ± 14.7 pts | 23 / 40 | 0 | log · contract · answers · summary | command |
| GPQA 30, thinking on 30 questions, limit 16,384 tokens | 80.0 % ± 13.9 pts median 1,954 tokens, 60.5 s per question | 24 / 30 | 5 | log · contract · answers · summary | command |
Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-10-24.
Note from the run: MMLU-Pro and GPQA Diamond ran with an answer limit of 48 tokens (my usual limit for these two tests, the answer is one letter). 49 of 400 MMLU-Pro answers and 46 of 198 GPQA Diamond answers were still cut off at that limit and count as wrong; they would need a longer limit to finish, and I have not re-run them with one. 2 of 40 LiveCodeBench answers were cut off at their limit of 32,768 tokens, 3 of 80 long-text answers at their limit of 128 tokens and 5 of 30 GPQA answers with thinking at the limit of 16,384 tokens; these count as wrong too. The questions of each test ran 4 at a time, the slot count of the recipe's server (same questions, limits and seed 42 for every question). Sampling: temperature 1.0, top_p 0.95, top_k 20 (the recipe). These are my own tests, and no other run of this setup is mixed into the numbers on this page.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-bertholomus-exl3-2.5bpw-tp2-2x-v072-vx.json --parts mmlupro --job bert-tp2-2410bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-bertholomus-exl3-2.5bpw-tp2-2x-v072-vx.json --parts gpqa --job bert-tp2-2410bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-bertholomus-exl3-2.5bpw-tp2-2x-v072-vx.json --parts ifeval --job bert-tp2-2410bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-bertholomus-exl3-2.5bpw-tp2-2x-v072-vx.json --parts lcb --job bert-tp2-2410bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-bertholomus-exl3-2.5bpw-tp2-2x-v072-vx.json --parts ruler --job bert-tp2-2410bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-bertholomus-exl3-2.5bpw-tp2-2x-v072-vx.json --parts ruler --job bert-tp2-2410bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-bertholomus-exl3-2.5bpw-tp2-2x-v072-vx.json --parts think --job bert-tp2-2410How much context fits
| Context size | Prompttokens | Filled | First token | Prefilltok/s, as the server reports it | Decodetok/s | Peak memoryGiB | Result |
|---|---|---|---|---|---|---|---|
| 4K 4,096 tokens | 3,891 | 95.0 % | — | — | — | 98.05 | answered (length) |
| 8K 8,192 tokens | 7,782 | 95.0 % | — | — | — | 98.10 | answered (length) |
| 16K 16,384 tokens | 15,564 | 95.0 % | — | — | — | 98.19 | answered (length) |
| 32K 32,768 tokens | 31,130 | 95.0 % | — | — | — | 98.29 | answered (stop) |
| 64K 65,536 tokens | 62,259 | 95.0 % | — | — | — | 98.46 | answered (stop) |
| 128K 131,072 tokens | 124,518 | 95.0 % | — | — | — | 98.67 | answered (stop) |
Memory test. The recipe fixes the context (1,048,576 tokens), so the server is started once with the recipe's own command and reserves its memory up front. Then, for each size (4K, 8K, 16K ... up to the largest size tested), one prompt that fills about 95 % of that size is sent, and the model must answer. The largest size that answered counts; it is the highest size tested, not a limit of the setup. Decode speed at that size was not measured. Peak memory is, on each of the two boxes, the rise of used memory (MemTotal - MemAvailable) over the idle value before the step; the table and the number show the highest of the two boxes. Unified memory, so GPU and CPU share it. Measured 2026-10-24. Raw files: result of every step · log · contract.
Note from the run: fit memory per node, highest rise over idle: box-3 98.7 GiB, box-2 96.9 GiB The recipe fixes the context (1,048,576 tokens, 4 requests at once), so the server is started once with the author's command and reserves its memory up front. Every size from 4K to 128K answered. The largest size tested is 131,072 tokens, the highest size in my list, not a limit of the setup: the window is 1,048,576. Decode speed at these sizes is not measured. The memory number is the rise of used memory over the idle value (unified memory, so GPU and CPU share it) and includes the model weights. The number shown (98.67 GiB) is the highest of the two GB10 boxes: 98.7 GiB on the first and 96.9 GiB on the second.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-bertholomus-exl3-2.5bpw-tp2-2x-v072-vx.json --parts fit --job bert-tp2-2410Also made this possible
- Model
DeepSeekHugging Face @deepseek_ai GitHub Website - Model files (EXL3 2.5 bpw, rev 7fbc8d00)bertholomusHugging Face @bertholomusai GitHub
- Engine (TensorFold)
Ash Hart@ashxhart GitHub Hugging Face Website - EXL3 format
turboderp@turboderp_ Hugging Face GitHub
Run it yourself
What you need and how to start it, step by step. The results below come from exactly this setup.
What you need
- Hardware
- 2x NVIDIA DGX Spark class boxes, tensor parallel 2 over two CX7 cables with RDMA: the first (rank 0) and the second (rank 1, an ASUS Ascent GX10), GPU clock capped at 2200 MHz. The ASUS GX10 runs an older driver (580.173.02) and kernel (6.17.0-1031-nvidia) than the first box (580.178.04, 7.0.0-1019-nvidia). That is named openly.
- Model
- bertholomus DeepSeek-V4.1-Flash-1017 EXL3 2.5bpw (rev 7fbc8d00) + checkpoint DSpark drafts
- Model files (EXL3 2.5 bpw, rev 7fbc8d00)
- huggingface.co/bertholomus/DeepSeek-V4.1-Flash-1017-EXL3-2.5bpw
- Engine
- TensorFold native engine, tensor parallel on 2 GB10 boxes, EXL3 2.5 bpw with DSpark drafts — the program that loads the model and answers requests · github.com/ashhart/TensorFold
- Recipe, EXL3 2.5 bpw checkpoint and DeepSeek-V4.1 support in TensorFold
- github.com/bertholomus/deepseek-v4.1-tensorfold-tp2-2xgb10 by bertholomus
Steps
Get the recipe, the engine and the model files from the list above.
Start the server with this command. It loads the model into memory.
bert-tp2-2x.sh start
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the start text exactly as our run recorded it, with the engine build in front, so anyone can repeat the result.
TensorFold native (Zig) a57deb9d, branch deepseek-v41-zig-tp2 (v0.7.2), TP=2 2D split (parity), DSpark drafts, RDMA devices rocep1s0f1+roceP2p1s0f1 (GID index 5, RoCE v2), kit of recipe 30a5bf9 (367 files); bert-tp2-2x.sh start (box-3; the recipe's scripts/serve.sh start with the ready config.env: preflight, page-cache drop once on both nodes, rank 1 tf-dsv41-lanes on box-2, rank 0 tensorfold-native serve on box-3, waits for /v1/models)Context and sampling: 1,048,576 tokens (the recipe's setting and the model window), a window of 2,097,152 positions shared by 4 streams and 4 requests at once, seed 42, no prompt cache. Sampling as in the recipe, the same with thinking on and off: temperature 1.0, top_p 0.95, top_k 20. The base model card says top_p 1.0; I kept the author's 0.95. Speed, quality and sparkDash with thinking off (GPQA 30 and the build test with thinking on). The recipe's server default of 32,768 tokens per answer applies only when a request sets none; I raised that default in my server config to 262,144, the limit the model card names, and the build test asked for 262,144 (named on this page); the quality tests set their own limits (named in the quality note). Agent for the build test: DeepSeek Harness 0.2.1-rc.2, headless.. Hardware: 2x NVIDIA DGX Spark class boxes, tensor parallel 2 over two CX7 cables with RDMA: the first (rank 0) and the second (rank 1, an ASUS Ascent GX10), GPU clock capped at 2200 MHz. The ASUS GX10 runs an older driver (580.173.02) and kernel (6.17.0-1031-nvidia) than the first box (580.178.04, 7.0.0-1019-nvidia). That is named openly.. All raw files of this setup: raw/v2/deepseek-v41-flash-2sparks-bertholomus-tp2.
Mia's own benchmark, run unchanged
This is Mia's own benchmark tool (commit 21a3511), run unchanged. It has its own section on this page, and it is never mixed with my numbers. A cut-off answer counts as wrong here, so a value can differ from sparkDash's own screen.
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| Quality QA 150 questions | 89.3 % ± 5.0 pts | 134 / 150 | 0 | summary | command |
| Quality MMLU 285 questions | 93.0 % ± 3.0 pts | 265 / 285 | 0 | summary | command |
| Tool Eval Quick 15 scenarios, tool use | 93 / 100 | 13 / 15 passed | — | tool eval result · tool eval report · tool eval command · summary | command |
| Decode, 1 request at once first token 106 ms | 155.3 tok/s | — | — | decode result · summary | command |
| Prefill at 4K first token 2.0 s | 2252.7 tok/s | — | — | prefill result · summary | command |
| Prefill at 8K first token 3.7 s | 2348.2 tok/s | — | — | prefill result · summary | command |
| Prefill at 16K first token 7.7 s | 2186.4 tok/s | — | — | prefill result · summary | command |
| Prefill at 32K first token 15.6 s | 2121.1 tok/s | — | — | prefill result · summary | command |
Seed 1234 applies to the two quality tests; decode, prefill and tool eval send no seed. The benchmark client ran on a separate machine that is never part of the measured group. Measured 2026-10-24. Version 2.50.0, commit 21a3511.
QA. Quality QA, 150 items, seed 1234, temperature 0, thinking off, 4 requests at once (Mia's default). An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
MMLU. Quality MMLU, 285 items (5 for each of 57 subjects, 4 answers), seed 1234, temperature 0, thinking off. An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
Decode. Decode bench, 1 request at once, 400 tokens, prompt type structured, temperature 0, thinking off. No seed is sent.
Prefill. Prefill bench at 4K, 8K, 16K and 32K tokens. sparkDash skips a size that does not fit into the context. No seed is sent.
Tool Eval Quick. Tool Eval Quick: tool-eval-bench (SeraphimSerapis, commit c61319b), 15 core scenarios (--short), --no-think, --timeout 120. Score 0 to 100. No seed is sent.
Note from the run: sparkDash runs its own tests (its own QA and MMLU questions at temperature 0, thinking off, a cut-off answer counts as wrong; none of the QA answers and none of the MMLU answers were cut off), so its decode and prefill values are not comparable with Speed v2 above: its decode test uses a structured prompt, Speed v2 a code and a prose prompt. The sparkDash client ran on another machine and was never part of the measured group.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 tools/quality2/sparkdash_client.py tools/quality2/cards/ds1017-bertholomus-exl3-2.5bpw-tp2-2x-v072-vx.json <box> --client <client machine> --job bert-tp2-2410 --out $RUNtool-eval-bench --base-url http://127.0.0.1:<port> --model DeepSeek-V4.1-Flash-1017-TP2 --short --no-think --timeout 120 --json-file $BENCH/sparkdash-client/runs/bert-tp2-2410-sparkdash-r17/tooleval/result.json --no-live