niklaslenzsudoingX's 2-Spark recipe: DeepSeek-V4.1-Flash-1017
2x NVIDIA DGX Spark (GB10, 128 GB unified memory each) linked by CX7 RDMA (about 109 Gb/s), GPU clock capped at 2200 MHz. Driver 580.173.02 on the first and 580.178.04 on the second GB10 box (the author tested 580.142 and 580.173.02); kernels 6.17.0-1031-nvidia and 7.0.0-1019-nvidia.
Overall score as of 23 Oct 2026. Scores are relative to the best run in the class today; a better model later moves older runs down.
How the overall score is calculated
| Part | This run | Best in class | Relative | Weight | Points |
|---|---|---|---|---|---|
| Speed, one request | decode 63.6 tok/s | best 68.5 tok/s | 92.7 % | 17.5 % | 16.2 |
| Speed, 8 at once | total 66.3 tok/s | best 99.6 tok/s | 66.5 % | 17.5 % | 11.6 |
| Quality | mean of 7 tests: 69.1 % | best mean 69.1 % | 100.0 % | 25 % | 25.0 |
| Browser check | 8.5 of 10 | 85.0 % | 5 % | 4.3 | |
| My rating | 6.5 of 10 | 65.0 % | 35 % | 22.8 | |
| Overall | 100 % | 79.9 |
The seven quality tests: MMLU-Pro 71.3 · GPQA 52.5 · IFEval 90.0 · LiveCodeBench 65.0 · long text 8K 70.0 · long text 32K 55.0 · GPQA 30 with thinking 80.0.
The speed part is half the decode speed with one request and half the total speed with 8 requests at once, each relative to the best run of this class; a run without a total for 8 requests at once gets the better of its decode speed alone and the mix with its highest measured total. Quality is relative to the best run of this class, the browser check and my rating are out of 10. The parts are rounded one by one. sparkDash, the memory test and speculative-decoding numbers are not part of it. How the score is made, and the assumption behind it.
What it builds as an agent
Task: Voxel world (3D scene, one HTML file). One attempt, no help. Counted: the final file counts.
The exact promptcopy it and try it yourself↓
This is the task prompt word for word, as we handed it to the agent harness. The harness adds its own system prompt and tools, and some harnesses add short context notes of their own. A run that goes past 90 minutes also gets a short “finish now” message, as the prompt says. To try it yourself, give it to your own coding agent: it needs a shell and a working folder.
Design and create the most impressive, creative, elaborate and detailed
voxel art world you are capable of. The theme and style are ENTIRELY your
choice — pick whatever world you can execute best and push it as far as
you can: scale, variety, detail, color, animation, atmosphere, life.
Surprise me.
Hard requirements:
- Everything ends up in ONE single self-contained HTML file (game.html in
the current working directory) that opens and runs directly in Chrome.
Use WebGL and whatever libraries you like, but embed everything —
no internet access at view time, no external files.
- The scene must present itself: include an automatic slow cinematic
camera tour (orbiting the world, occasionally zooming in on highlights)
that starts on load, plus mouse controls (orbit/zoom) for manual viewing.
- All user-facing text (UI, HUD, titles, descriptions, hints) must be in
ENGLISH, regardless of the language of any project files you encounter.
When finished, verify your HTML actually loads and renders without
console errors, then state DONE with the absolute path of game.html.
Do not ask questions.
How you work (same for every model):
- You work alone in the folder /work (this is the current working directory). Nobody will answer questions. Aim to finish in about 60 minutes. After 90 minutes you will get a message to finish; then you have at most 30 more minutes. Hard stop after 120 minutes.
- Tools on this machine: shell, files, python3 with numpy and Pillow, ffmpeg, node, and `shot.sh game.html` (opens the page in a headless browser, saves a screenshot and prints console errors). There is no internet and no access to other machines.
- Keep a working /work/game.html at all times. If you are stopped, the last version that loaded without errors counts.
- JavaScript libraries available offline in the test browser, through an import map with these exact URLs: "three": "https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.module.js", "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/", "simplex-noise": "https://cdn.jsdelivr.net/npm/simplex-noise@4.0.3/dist/esm/simplex-noise.js", "@tweenjs/tween.js": "https://cdn.jsdelivr.net/npm/@tweenjs/tween.js@25.0.0/dist/tween.esm.js", "cannon-es": "https://cdn.jsdelivr.net/npm/cannon-es@0.20.0/dist/cannon-es.js". No other external resources.
- After the scene is built, set `window.scene` to the Three.js scene (if you use Three.js) and `window.ready = true`.
Play it yourself
Open full screenAI-generated The page this setup built, running live in your browser: click into it and use the controls it shows itself (usually drag to look around, scroll to zoom). It needs WebGL and can be slow on a weak graphics card.
The browser check
- page loads without page error1 / 1
- window.ready === true within 30 s1 / 1
- no console errors (whole session)1 / 1
- frame is not blank1 / 1
- frame rate in software GL (rAF over 5 s)0.5 / 1
- world: scene size 0.5 (parts >= 80 or >= 20000 triangles; canvas fallback), visible text English if any 0.51 / 1
- automatic camera tour: the view changes on its own in two successive ~5 s windows (1 each)2 / 2
- animation in the world (nodes / instances / particles moving besides the camera)1 / 1
- OrbitControls: drag and wheel change the picture beyond the animation baseline0 / 1
Measured 2026-10-23. Raw files: check · run info · command · final choice · contract · sampling · gate stats · gate log · screenshot · clip.mp4.
Other processes used up to 133 % CPU during the run. Browser check 8.5 out of 10 (voxel-world task): the page loads, runs without console errors, is not blank, shows a world (143 parts, 322,302 triangles, English text), flies an automatic camera tour in both windows and animates; the 1.5 lost points are the frame rate (14.6 frames per second in software rendering without a GPU, 20 are needed: half a point) and the orbit check (mouse drag and wheel changed the picture, but not beyond what the page animates on its own: 1 point). The agent (OpenCode 1.19.3 with my launcher that sets the model's input limit to its context; the author names no agent harness that runs locally, so OpenCode) finished by itself after 49.3 minutes, one try, no help; the “finish now” message (from 90 minutes) was not needed. 17 versions of the page were seen, 15 loaded without errors, and the last one counts. 68 requests, no error, no request blocked (7,647,347 prompt tokens, 125,806 answer tokens); the longest answer had 56,097 tokens and none ended at the limit. The limit per answer was 262,144 tokens, the value on the DeepSeek model card. The author's server rejects any request whose prompt plus answer limit is larger than its 262,144-token window (HTTP 400, it does not clip), so my test gate lowered the limit in every request to the room left in the context minus a safety margin (259,709 in the first request; 64,128 in the last one: its prompt had 183,144 tokens, and the gate took the estimate of 183,111 plus a safety margin of 14,905 tokens off the 262,144); I name this because it is lower than the card value. Thinking was on, sampling temperature 1.0 and top_p 1.0. A first start of this test (show-r4) never started the agent because a harness setting was missing in my test script; it produced no model output, and show-r5 is the only agent run.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
run_showcase.sh --harness opencode-limit-input --task VX --upstream http://127.0.0.1:18080/v1 --model dsv41 --sampling-file $RUN/quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3-show-r5/show.in/sampling-show.json --out $RUN/quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3-show-r5/show --limit-s 7200 --inject-after-s 5400 --poll-s 20 --ctx 262144How fast it runs
| Measurement | Result |
|---|---|
| Decode | 63.6 tok/s |
| Words per second | 43.2 words/s |
| Characters per second | 277.6 chars/s |
| Decode, code prompt | 64.9 tok/s |
| Decode, prose prompt | 62.2 tok/s |
| Spread of the decode runs (CV) | 8.3 % |
| Spread of the code runs (CV) | 8.3 % |
| Spread of the prose runs (CV) | 0.8 % |
| Decode, window 1 | 65.1 tok/s |
| Prefill at 2K | 1710.0 tok/s |
| Prefill at 8K | 1815.0 tok/s |
| Prefill at 32K | 1796.9 tok/s |
| Time to first token at 2K | 1.2 s |
| Time to first token at 8K | 4.5 s |
| Time to first token at 32K | 18.2 s |
| Draft acceptance rate | 31.9 % |
| Mean accepted length | 2.6 tokens |
| 1 request at once (total) | 51.3 tok/s |
| 2 at once (total) | 65.6 tok/s |
| 4 at once (total) | 66.2 tok/s |
| 8 at once (total) | 66.3 tok/s |
| 1 request at once (per request) | 51.3 tok/s |
| 2 at once (per request) | 33.7 tok/s |
| 4 at once (per request) | 24.0 tok/s |
| 8 at once (per request) | 13.6 tok/s |
| Server slots (from the recipe) | 2 slots |
| Long answer, wall time | 69.9 s |
Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-10-23. Raw files: speed result · speed contract · log · long answer.
Requests at once: the server has 2 slots (from the author's recipe). C4, C8 exceed the slots: the extra requests wait in the queue, and they were measured anyway. The speed test ran with the author's launcher (release v2.0, commit f8f20b3): vLLM with DSpark drafting (k=5), tensor parallel over two GB10 boxes (CX7 RDMA, about 109 Gb/s), 2 sequences at a time (MAX_SEQS 2), context 262,144 tokens per request, the Engram tables (203 GB) read from NVMe at lookup time (the author's setting). The author names no sampling for serving, so the DeepSeek model card applies: temperature 1.0, top_p 1.0 (top_k not set); seed 42; the request flag for no thinking. The headline 63.55 tok/s is the mean of the code prompt (64.87) and the prose prompt (62.23), medians of two runs each (spread 8.3 % and 0.8 %). Drafts were accepted at a rate of 31.9 % (mean accepted length 2.6 tokens). The server has 2 slots, so 4 and 8 requests at once exceed them: the extra requests wait in the queue and were measured anyway; the total stays near 66 tok/s. The author's own speed numbers use greedy decoding (temperature 0); my Speed v2 uses the model card's sampling, so they are not mixed. Three adaptations for my hardware, none of them a tuning: my CX7 cable joins port 0 of one GB10 box to port 1 of the other, so the launcher's network-interface setting is given per node (without it the launcher uses the author's single setting); a command for the node the script runs on goes through bash instead of ssh; the container image was copied to the second GB10 box by hand and the image IDs were compared. Every server setting (context, slots, memory, drafting, Engram offload, parsers) is the author's default.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 speed2.py --base-url http://127.0.0.1:40873/v1 --label quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3-speed-r0/quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3-speed-r0/speed2 --contract $RUN/quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3-speed-r0/quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1 --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --tokenizer-json $HOME/models/dsv41-exl3/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 1.0, "top_p": 1.0} --sampling-source Author (sudoingX): names no serving sampling (README.md, GUIDE.md, pack README: only 'greedy' for his own benchmark methods; the launcher sets no sampling flag). Therefore the base model card applies: deepseek-ai/DeepSeek-V4.1-Flash-1017 @1bae0e10 README.md:77-84 'Recommended sampling parameters: temperature 1.0, top_p 1.0, context_window 1M tokens, max_tokens >= 256K'; top_k not named -> not set. Same setting for thinking and non-thinking. Speed v2 runs with thinking off (enable_thinking false per request), think/show with thinking on. NOTE: the author's own speed numbers (kit_bench.py set b) are greedy (temperature 0); our Speed v2 uses the card sampling (SENDEPLAN 7.3); the author's greedy method may run as an extra appendix row, never mixed. --model dsv41 --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 2 --server-slots-source sudoingx/dgx-spark-deepseek-v41@f8f20b3 scripts/launch/dsv41_tp2.sh:19 (MAX_SEQS=2, --max-num-seqs); README.md:136 ('Two sequences at a time (MAX_SEQS=2)'). C4 and C8 exceed the 2 slots and measure the queue by design (exceeds_slots).How well it answers
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| MMLU-Pro 400 fixed questions, letter only | 71.3 % ± 4.4 pts | 285 / 400 | 4 | summary · log · contract | command |
| GPQA Diamond 198 questions, letter only | 52.5 % ± 6.9 pts | 104 / 198 | 16 | answers · summary · log · contract | command |
| IFEval 200 fixed prompts, strict | 90.0 % ± 4.2 pts | 180 / 200 | 0 | answers · summary · log · contract | command |
| LiveCodeBench 40 newest tasks, the code runs | 65.0 % ± 14.2 pts | 26 / 40 | 0 | summary · log · contract | command |
| Long text 8K 4 tasks x 10 at 8192 tokens | 70.0 % ± 13.7 pts | 28 / 40 | 1 | log · contract · answers · summary | command |
| Long text 32K 4 tasks x 10 at 32768 tokens | 55.0 % ± 14.7 pts | 22 / 40 | 2 | log · contract · answers · summary | command |
| GPQA 30, thinking on 30 questions, limit 16,384 tokens | 80.0 % ± 13.9 pts median 1,365 tokens, 20.1 s per question | 24 / 30 | 5 | log · contract · answers · summary | command |
Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-10-23.
Note from the run: MMLU-Pro and GPQA Diamond ran with an answer limit of 1,024 tokens. In some answers the model explains before it gives the letter: in my first run with the usual 48-token limit, 134 of 598 answers were cut off in the middle of the explanation and counted as wrong. I ran both tests again with 1,024 tokens (same questions, seed 42, same sampling and setup). This page shows only the 1,024-token values and never mixes them with the 48-token run. Even at 1,024 tokens, 4 of 400 MMLU-Pro answers and 16 of 198 GPQA answers were cut off and count as wrong. 3 of the 80 long-text answers (1 at 8K, 2 at 32K) and 5 of 30 GPQA answers with thinking (limit 16,384 tokens) were cut off too. GPQA Diamond ran on the public mirror fingertap/GPQA-Diamond (the original is gated). The author's pack README gives a quantisation loss against the original model (KLD 0.0198); my suites measure the quality separately. These are my own tests.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3.json --parts mmlupro --job quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3-mcq1024bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3.json --parts gpqa --job quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3-mcq1024bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3.json --parts ifeval --job quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3.json --parts lcb --job quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3.json --parts ruler --job quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3.json --parts ruler --job quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3.json --parts think --job quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3How much context fits
| Context size | Prompttokens | Filled | First token | Prefilltok/s, as the server reports it | Decodetok/s | Peak memoryGiB | Result |
|---|---|---|---|---|---|---|---|
| 4K 4,096 tokens | 3,891 | 95.0 % | — | — | — | 111.75 | answered (length) |
| 8K 8,192 tokens | 7,782 | 95.0 % | — | — | — | 111.75 | answered (stop) |
| 16K 16,384 tokens | 15,564 | 95.0 % | — | — | — | 111.74 | answered (stop) |
| 32K 32,768 tokens | 31,130 | 95.0 % | — | — | — | 111.84 | answered (stop) |
| 64K 65,536 tokens | 62,259 | 95.0 % | — | — | — | 112.10 | answered (stop) |
| 128K 131,072 tokens | 124,518 | 95.0 % | — | — | — | 112.16 | answered (stop) |
Memory test. The recipe fixes the context (262,144 tokens), so the server is started once with the recipe's own command and reserves its memory up front. Then, for each size (4K, 8K, 16K ... up to the largest size tested), one prompt that fills about 95 % of that size is sent, and the model must answer. The largest size that answered counts; it is the highest size tested, not a limit of the setup. Decode speed at that size was not measured. Peak memory is, on each of the two boxes, the rise of used memory (MemTotal - MemAvailable) over the idle value before the step; the table and the number show the highest of the two boxes. Unified memory, so GPU and CPU share it. Measured 2026-10-23. Raw files: result of every step · log · contract.
Note from the run: fit memory per node, highest rise over idle: box-2 112.2 GiB, box-1 110.7 GiB The recipe fixes the context (262,144 tokens, 2 sequences), so the server is started once with the author's command and reserves its memory up front. Every size from 4K to 128K answered. The largest size tested is 131,072 tokens, the highest size in the list, not a limit of the setup: the author did not run the 256K rung for v2.0 either. The memory number is, on each of the two GB10 boxes, the rise of used memory over the idle value (unified memory, so GPU and CPU share it); the number shown is the higher of the two (112.16 GiB; the other GB10 box peaked at 110.7 GiB). Decode speed at these sizes was not measured.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3.json --parts fit --job quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3Also made this possible
- Model
DeepSeekHugging Face @deepseek_ai GitHub Website - EXL3 2.8 bpw weightsSsudoingx/DeepSeek-V4.1-Flash-1017-EXL3-2.8bpwHugging Face
- EXL3 format and kernels
turboderp@turboderp_ Hugging Face GitHub - Designs credited in the recipe's NOTICE (TensorFold)
Ash Hart@ashxhart GitHub Hugging Face Website
Run it yourself
What you need and how to start it, step by step. The results below come from exactly this setup.
What you need
- Hardware
- 2x NVIDIA DGX Spark (GB10, 128 GB unified memory each) linked by CX7 RDMA (about 109 Gb/s), GPU clock capped at 2200 MHz. Driver 580.173.02 on the first and 580.178.04 on the second GB10 box (the author tested 580.142 and 580.173.02); kernels 6.17.0-1031-nvidia and 7.0.0-1019-nvidia.
- Model
- DeepSeek-V4.1-Flash-1017 EXL3 2.8bpw (sudoingX, HF rev aba08ea0) + DSpark drafter (in the pack) + Engram tables from deepseek-ai rev 1bae0e10
- EXL3 2.8 bpw weights
- huggingface.co/sudoingx/DeepSeek-V4.1-Flash-1017-EXL3-2.8bpw
- Engine
- vLLM, tensor parallel on 2 GB10 boxes, EXL3 2.8 bpw with DSpark drafts (release v2.0 of 22 Oct 2026) — the program that loads the model and answers requests
- Recipe, EXL3 quant (2.8 bpw), vLLM patches and DSpark drafter
- github.com/sudoingx/dgx-spark-deepseek-v41 by sudoingX
Steps
Get the recipe and the model files from the list above. The list has no link for the engine (vLLM, tensor parallel on 2 GB10 boxes, EXL3 2.8 bpw with DSpark drafts (release v2.0 of 22 Oct 2026)).
Start the server with this command in the recipe folder. It loads the model into memory.
HEAD=box-2 \ WORKER=box-1 \ FAB_IF_HEAD=enp1s0f1np1 \ FAB_HCA_HEAD=rocep1s0f1 \ FAB_IF_WORKER=enp1s0f0np0 \ FAB_HCA_WORKER=rocep1s0f0 \ MODEL_DIR=$HOME/models/dsv41-exl3 ./scripts/launch/dsv41_tp2-pernode.sh upWe used the recipe at commit f8f20b3.
What the parts mean
MODEL_DIR=$HOME/models/dsv41-exl3the folder with the model files
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the start text exactly as our run recorded it, with the engine build in front, so anyone can repeat the result.
upstream vLLM 6e517b15 (0.31.1.dev0+g6e517b15.dsv41gb10) + sudoingX patches 0001-0008, 0010-0021, 0023-0038 + plugin vllm_exl3_moe (exl3_smoe, exl3_mrow kernels, RDMA all-reduce ring), exllamav3 v1.6.0, TP=2 over RoCE v2, DSpark speculative decoding (k=5, per-round depth), image dsv41-serve:v5; HEAD=box-2 WORKER=box-1 FAB_IF_HEAD=enp1s0f1np1 FAB_HCA_HEAD=rocep1s0f1 FAB_IF_WORKER=enp1s0f0np0 FAB_HCA_WORKER=rocep1s0f0 MODEL_DIR=$HOME/models/dsv41-exl3 ./scripts/launch/dsv41_tp2-pernode.sh up ($HOME/dsv41_recipe @f8f20b3; via spark-2x-dsv41-sudoingx.sh start; the author's command is the same script with one FAB_IF/FAB_HCA for both nodes)Context and sampling: 262,144 tokens per request (the author's launcher default; the model window is 1M tokens, which does not fit with DSpark on), 2 sequences at a time, seed 42, requests ask for no prompt cache. Sampling as the model card gives it, because the author names none: temperature 1.0, top_p 1.0 (one set for thinking on and off). Speed and the quality tests with thinking off (GPQA 30: on); the build test with thinking on. Answer limit for the build test 262,144 tokens (the model card), lowered by my test gate to the room left in the context. Agent for the build test: OpenCode 1.19.3. Three adaptations for my hardware, none of them a tuning: my CX7 cable joins port 0 of one GB10 box to port 1 of the other, so the launcher's network-interface setting is given per node (without it the launcher uses the author's single setting); a command for the node the script runs on goes through bash instead of ssh; the container image was copied to the second GB10 box by hand and the image IDs were compared. Every server setting (context, slots, memory, drafting, Engram offload, parsers) is the author's default. . Hardware: 2x NVIDIA DGX Spark (GB10, 128 GB unified memory each) linked by CX7 RDMA (about 109 Gb/s), GPU clock capped at 2200 MHz. Driver 580.173.02 on the first and 580.178.04 on the second GB10 box (the author tested 580.142 and 580.173.02); kernels 6.17.0-1031-nvidia and 7.0.0-1019-nvidia.. All raw files of this setup: raw/v2/deepseek-v41-flash-2sparks-sudoingx-vllm.
Mia's own benchmark, run unchanged
This is Mia's own benchmark tool (commit 21a3511), run unchanged. It has its own section on this page, and it is never mixed with my numbers. A cut-off answer counts as wrong here, so a value can differ from sparkDash's own screen.
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| Quality QA 150 questions | 92.0 % ± 4.4 pts | 138 / 150 | 1 | summary | command |
| Quality MMLU 285 questions | 93.0 % ± 3.0 pts | 265 / 285 | 0 | summary | command |
| Tool Eval Quick 15 scenarios, tool use | 90 / 100 | 13 / 15 passed | — | tool eval result · tool eval report · tool eval command · summary | command |
| Decode, 1 request at once first token 214 ms | 141.6 tok/s | — | — | decode result · summary | command |
| Prefill at 4K first token 2.1 s | 2228.5 tok/s | — | — | prefill result · summary | command |
| Prefill at 8K first token 4.2 s | 2096.0 tok/s | — | — | prefill result · summary | command |
| Prefill at 16K first token 8.1 s | 2082.7 tok/s | — | — | prefill result · summary | command |
| Prefill at 32K first token 18.4 s | 1806.6 tok/s | — | — | prefill result · summary | command |
Seed 1234 applies to the two quality tests; decode, prefill and tool eval send no seed. The benchmark client ran on a separate machine that is never part of the measured group. Measured 2026-10-23. Version 2.50.0, commit 21a3511.
QA. Quality QA, 150 items, seed 1234, temperature 0, thinking off, 4 requests at once (Mia's default). An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
MMLU. Quality MMLU, 285 items (5 for each of 57 subjects, 4 answers), seed 1234, temperature 0, thinking off. An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
Decode. Decode bench, 1 request at once, 400 tokens, prompt type structured, temperature 0, thinking off. No seed is sent.
Prefill. Prefill bench at 4K, 8K, 16K and 32K tokens. sparkDash skips a size that does not fit into the context. No seed is sent.
Tool Eval Quick. Tool Eval Quick: tool-eval-bench (SeraphimSerapis, commit c61319b), 15 core scenarios (--short), --no-think, --timeout 120. Score 0 to 100. No seed is sent.
Note from the run: sparkDash runs its own tests (its own QA and MMLU questions at temperature 0, thinking off, a cut-off answer counts as wrong; 1 of the QA answers was cut off, none of the MMLU answers), so its decode and prefill values are not comparable with Speed v2 above. The sparkDash client ran on another machine and was never part of the measured group.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 tools/quality2/sparkdash_client.py tools/quality2/cards/ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3.json <box> --client <client machine> --job quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3 --out $RUNtool-eval-bench --base-url http://127.0.0.1:<port> --model dsv41 --short --no-think --timeout 120 --json-file $BENCH/sparkdash-client/runs/quality-v2-261022-box-2-ds1017-sudoingx-exl3-2.8bpw-v5-2x-f8f20b3-sparkdash-r7/tooleval/result.json --no-live