niklaslenzMia's 2-Spark recipe: DeepSeek-V4.1-Flash-1017
2x NVIDIA GB10 boxes linked by CX7 RDMA (about 109 Gb/s): 1x Gigabyte AI TOP ATOM and 1x Lenovo ThinkStation PGX. Both run driver 580.178.04 and kernel 7.0.0-1019-nvidia, GPU clock capped at 2200 MHz.
Overall score as of 20 Oct 2026. This is the only complete run in the class so far, so it is rank 1. Scores are relative to the best run in the class today; a better model later moves older runs down.
How the overall score is calculated
| Part | This run | Best in class | Relative | Weight | Points |
|---|---|---|---|---|---|
| Speed | decode 67.4 tok/s | best 67.4 tok/s | 100.0 % | 45 % | 45.0 |
| Quality | mean of 7 tests: 68.2 % | best mean 68.2 % | 100.0 % | 30 % | 30.0 |
| Browser check | 8 of 10 | 80.0 % | 5 % | 4.0 | |
| My rating | 7.5 of 10 | 75.0 % | 20 % | 15.0 | |
| Overall | 100 % | 94.0 |
The seven quality tests: MMLU-Pro 66.8 · GPQA 38.9 · IFEval 88.5 · LiveCodeBench 70.0 · long text 8K 67.5 · long text 32K 62.5 · GPQA 30 with thinking 83.3.
Speed and quality are relative to the best run of this class, the browser check and my rating are out of 10. The parts are rounded one by one. sparkDash, the memory test and speculative-decoding numbers are not part of it. How the score is made, and the assumption behind it.
What it builds as an agent
Task: Voxel world (3D scene, one HTML file). One attempt, no help. Counted: the final file counts.
Play it yourself
Open full screenAI-generated The page this setup built, running live in your browser: click into it and use the controls it shows itself (usually drag to look around, scroll to zoom). It needs WebGL and can be slow on a weak graphics card.
The browser check
- page loads without page error1 / 1
- window.ready === true within 30 s1 / 1
- no console errors (whole session)1 / 1
- frame is not blank1 / 1
- frame rate in software GL (rAF over 5 s)0 / 1
- world: scene size 0.5 (parts >= 80 or >= 20000 triangles; canvas fallback), visible text English if any 0.51 / 1
- automatic camera tour: the view changes on its own in two successive ~5 s windows (1 each)2 / 2
- animation in the world (nodes / instances / particles moving besides the camera)1 / 1
- OrbitControls: drag and wheel change the picture beyond the animation baseline0 / 1
Measured 2026-10-20. Raw files: check · run info · command · final choice · contract · sampling · screenshot · clip.mp4.
Other processes used up to 61 % CPU during the run. Browser check 8.0 out of 10: the page loads, runs without console errors, is not blank, shows a world, flies an automatic camera tour and animates; the two lost points are the frame rate (3.0 frames per second in software rendering without a GPU, 20 are needed) and the orbit check (mouse drag and wheel changed the picture less than the check needs). The server's default of 32,768 tokens per answer (recipe, scripts/config.sh:106) was lifted to the card's 262,144 for every request; the longest answer in this run had 32,476 tokens. My test gate sent an empty keep-alive comment every 30 seconds so that the agent's connection was not cut after 300 seconds of silence; it changes nothing about model, prompt, sampling or output. The agent ended by itself after 59.7 minutes; the “finish now” message (at 90 minutes) was not needed. This is one run with working technique. Two earlier starts of the build test did not count because of my own tools, not the model: the first used the recipe's 32,768-token default and ended at that limit without a page, the second was cut by the 300-second silence abort of the agent client.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
run_showcase.sh --harness dsh-021 --task VX --upstream http://127.0.0.1:18080/v1 --model DeepSeek-V4.1-Flash-1017 --sampling-file $RUN/raw/plan-2010-show-r7/show.in/sampling-show.json --out $RUN/raw/plan-2010-show-r7/show --limit-s 7200 --inject-after-s 5400 --poll-s 20 --ctx 524288How fast it runs
| Measurement | Result |
|---|---|
| Decode | 67.4 tok/s |
| Words per second | 44.3 words/s |
| Characters per second | 292.9 chars/s |
| Spread of the decode runs (CV) | 4.9 % |
| Decode, window 1 | 68.1 tok/s |
| Prefill at 2K | 1698.3 tok/s |
| Prefill at 8K | 2046.1 tok/s |
| Prefill at 32K | 2126.7 tok/s |
| Time to first token at 2K | 1.2 s |
| Time to first token at 8K | 4.0 s |
| Time to first token at 32K | 15.4 s |
| Draft acceptance rate | 59.1 % |
| Mean accepted length | 2.4 tokens |
| 1 request at once (total) | 59.8 tok/s |
| Long answer, wall time | 41.8 s |
| Decode, code prompt | 73.0 tok/s |
| Decode, prose prompt | 61.7 tok/s |
| Spread of the code runs (CV) | 4.9 % |
| Spread of the prose runs (CV) | 1.1 % |
Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median. Measured 2026-10-20. Raw files: speed result · speed contract · log · long answer.
Decode with 2 and 4 requests at once is not measured: the tool could not read the server's slot count, although the recipe runs 4 streams. The prompt for code ran at 73.0 tok/s, the prose prompt at 61.7 tok/s (medians of two runs each); the headline 67.35 tok/s is the mean of the two. Speculative drafts were accepted at a rate of 59.1 %.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 speed2.py --base-url http://127.0.0.1:40153/v1 --label run-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/plan-2010-speed-r0/work/speed2 --contract $RUN/raw/plan-2010-speed-r0/work/speed2/speed-contract.json --parts decode,prefill,conc,l1 --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --tokenizer-json $HOME/hf-ds1017/hub/models--Mia-AiLab--DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw/snapshots/6d54be4fca338d79b17c3b03cc9a423bb3ac6718/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 1.0, "top_p": 0.95, "top_k": 20} --sampling-source Same setting for thinking and non-thinking (no separate profile): Mia's recipe server defaults scripts/config.sh:112-114 @e65785cd, same values and lines as @1afc891 (TEMPERATURE 1.0, TOP_P 0.95, TOP_K 20; 'the family's measured sampling, BertholomusAI two-node launch'; a request's own values win), recipe README.md:280 @e65785cd (:267 @1afc891) and the EAQ model card Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw README.md:82 ('Official sampling from the base card: temperature 1.0, top_p 0.95'). Note: the 1017 base card deepseek-ai/DeepSeek-V4.1-Flash-1017 README.md:77-84 recommends top_p 1.0; Mia's recipe uses 0.95; kept at 0.95 as in Mia's recipe (decision of 19 Oct). Speed v2 runs with thinking off (enable_thinking false per request), show/think with thinking on. --model DeepSeek-V4.1-Flash-1017 --no-cache-prompt --server-log $RUN/logs/server-A.log --engine-version-file $RUN/logs/engine-version.txtHow well it answers
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| MMLU-Pro 400 fixed questions, letter only | 66.8 % ± 4.6 pts | 267 / 400 | 47 | answers · summary · log · contract | command |
| GPQA Diamond 198 questions, letter only | 38.9 % ± 6.7 pts | 77 / 198 | 61 | answers · summary · log · contract | command |
| IFEval 200 fixed prompts, strict | 88.5 % ± 4.4 pts | 177 / 200 | 0 | summary · log · contract | command |
| LiveCodeBench 40 newest tasks, the code runs | 70.0 % ± 13.7 pts | 28 / 40 | 4 | summary · log · contract | command |
| Long text 8K 4 tasks x 10 at 8192 tokens | 67.5 % ± 14.0 pts | 27 / 40 | 3 | log · contract · answers · summary | command |
| Long text 32K 4 tasks x 10 at 32768 tokens | 62.5 % ± 14.4 pts | 25 / 40 | 0 | log · contract · answers · summary | command |
| GPQA 30, thinking on 30 questions, limit 16,384 tokens | 83.3 % ± 13.1 pts median 2,128 tokens, 31.4 s per question | 25 / 30 | 4 | log · contract · answers · summary | command |
Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-10-20.
Note from the run: 47 of 400 MMLU-Pro answers and 61 of 198 GPQA Diamond answers were cut off at the 48-token limit (the model often starts with an explanation instead of the letter) and count as wrong. 4 of 40 LiveCodeBench answers were cut off at their limit of 32,768 tokens, 3 of 80 long-text answers at their limit, and 4 of 30 GPQA answers with thinking at the limit of 16,384 tokens; these count as wrong too. Mia's model card says she has not measured the quality of this compressed checkpoint, so these numbers are my own tests.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-exl3-eaq-2.9bpw-2x.json --parts mmlupro --job plan-2010bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-exl3-eaq-2.9bpw-2x.json --parts gpqa --job plan-2010bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-exl3-eaq-2.9bpw-2x.json --parts ifeval --job plan-2010bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-exl3-eaq-2.9bpw-2x.json --parts lcb --job plan-2010bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-exl3-eaq-2.9bpw-2x.json --parts ruler --job plan-2010bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-exl3-eaq-2.9bpw-2x.json --parts ruler --job plan-2010bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-exl3-eaq-2.9bpw-2x.json --parts think --job plan-2010How much context fits
| Context size | Prompttokens | Filled | First token | Prefilltok/s, as the server reports it | Decodetok/s | Peak memoryGiB | Result |
|---|---|---|---|---|---|---|---|
| 4K 4,096 tokens | 3,891 | 95.0 % | — | — | — | 109.73 | answered (stop) |
| 8K 8,192 tokens | 7,782 | 95.0 % | — | — | — | 109.81 | answered (length) |
| 16K 16,384 tokens | 15,564 | 95.0 % | — | — | — | 109.88 | answered (stop) |
| 32K 32,768 tokens | 31,130 | 95.0 % | — | — | — | 110.02 | answered (length) |
| 64K 65,536 tokens | 62,259 | 95.0 % | — | — | — | 110.11 | answered (stop) |
| 128K 131,072 tokens | 124,518 | 95.0 % | — | — | — | 110.37 | answered (stop) |
Memory test. The recipe fixes the context (524,288 tokens), so the server is started once with the recipe's own command and reserves its memory up front. Then, for each size (4K, 8K, 16K ... up to the largest size tested), one prompt that fills about 95 % of that size is sent, and the model must answer. The largest size that answered counts; it is the highest size tested, not a limit of the setup. Decode speed at that size was not measured. Peak memory is, on each of the two boxes, the rise of used memory (MemTotal - MemAvailable) over the idle value before the step; the table and the number show the highest of the two boxes. Unified memory, so GPU and CPU share it. Measured 2026-10-20. Raw files: result of every step · log · contract.
Note from the run: fit memory per node, highest rise over idle: box-3 110.4 GiB, box-4 109.0 GiB The list of sizes ends at 128K (131,072 tokens), so this is the largest size tested, not the largest size the setup can load: the recipe runs at a context of 524,288. Decode speed at these sizes is not measured (the server returns no timings). The memory number is the rise over the idle value, so it includes the model weights; the number shown is the higher of the two boxes (the other one peaked at 109.05 GiB).
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/ds1017-exl3-eaq-2.9bpw-2x.json --parts fit --job plan-2010Also made this possible
- Model
DeepSeekHugging Face @deepseek_ai GitHub Website - Engine (TensorFold)
Ash Hart@ashxhart GitHub Hugging Face Website - EXL3 format
turboderp@turboderp_ Hugging Face GitHub
Run it yourself
What you need and how to start it, step by step. The results below come from exactly this setup.
What you need
- Hardware
- 2x NVIDIA GB10 boxes linked by CX7 RDMA (about 109 Gb/s): 1x Gigabyte AI TOP ATOM and 1x Lenovo ThinkStation PGX. Both run driver 580.178.04 and kernel 7.0.0-1019-nvidia, GPU clock capped at 2200 MHz.
- Model
- Mia DeepSeek-V4.1-Flash-1017 EXL3 EAQ v1 2.9bpw (rev 6d54be4f) + checkpoint DSpark drafts
- Engine
- TensorFold, tensor parallel on 2 GB10 boxes, EXL3 EAQ 2.9 bpw with DSpark drafts — the program that loads the model and answers requests · github.com/ashhart/TensorFold
- Recipe and EXL3 quant (EAQ 2.9 bpw)
- github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks by Mia (MiaAI-Lab)
Steps
Get the recipe and the engine from the list above. The list has no direct download link for the model file itself. The base model is huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017.
Start the server with this command in the recipe folder. It loads the model into memory.
PATH=$HOME/.local/bin HF_HUB_OFFLINE=1 ./start.shWe used the recipe at commit e65785cd. Our run started it on the first DGX Spark (head); the second DGX Spark ran the worker. How the recipe starts the two DGX Sparks: see the README of the recipe, github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the start text exactly as our run recorded it, with the engine build in front, so anyone can repeat the result.
TensorFold native 1.0.2 (Zig 0.17.0), TP=2, DSpark speculative drafts (DRAFTS=1), in image tensorfold-dsv41:exl3-tp2-8eb5919ff0a4 (TF_REF 629c524a + zig/patches 0001, 0002, 0004, 0005 prefill interleave 250 ms, 0006 image prompt cache; CHANGELOG.md:5-12 @e65785cd; tensorfold-native 1.0.2 per README.md:38); PATH=$HOME/.local/bin HF_HUB_OFFLINE=1 ./start.sh ($HOME/dsv41-1017-2x @e65785cd, scripts/local.sh: CONTEXT=524288; head box-3 rank 0, worker box-4 rank 1; via spark-2x-ds1017-eaq.sh start)Context and sampling: 524,288 tokens (Mia's own measurement condition for this recipe; the recipe default and the model window are 1,048,576), 4 streams share the pool, seed 42, no prompt cache. Sampling as in Mia's recipe, the same with thinking on and off: temperature 1.0, top_p 0.95, top_k 20. The base model card says top_p 1.0; I kept Mia's 0.95 because that is how she runs it. Speed and quality tests with thinking off; GPQA 30 and the build test with thinking on. The recipe's server default of 32,768 tokens per answer applies only when a request sets none; my requests set 262,144, the limit the model card names (named on this page). Agent for the build test: DeepSeek Harness 0.2.1-rc.2, headless. My test gate sends an empty keep-alive comment every 30 seconds while the model writes a long tool call, because the agent client aborts a stream after 300 seconds of silence; model, prompt, sampling and output are unchanged.. Hardware: 2x NVIDIA GB10 boxes linked by CX7 RDMA (about 109 Gb/s): 1x Gigabyte AI TOP ATOM and 1x Lenovo ThinkStation PGX. Both run driver 580.178.04 and kernel 7.0.0-1019-nvidia, GPU clock capped at 2200 MHz.. All raw files of this setup: raw/v2/deepseek-v41-flash-2sparks-mia-tensorfold.
Mia's own benchmark, run unchanged
This is Mia's own benchmark tool (commit 21a3511), run unchanged. It has its own section here and its own column on the class page, and it is never mixed with my numbers. A cut-off answer counts as wrong here, so a value can differ from sparkDash's own screen.
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| Quality QA 150 questions | 94.0 % ± 3.9 pts | 141 / 150 | 0 | summary | command |
| Quality MMLU 285 questions | 91.9 % ± 3.2 pts | 262 / 285 | 0 | summary | command |
| Tool Eval Quick 15 scenarios, tool use | 93 / 100 | 13 / 15 passed | — | tool eval result · tool eval report · tool eval command · summary | command |
| Decode, 1 request at once first token 115 ms | 129.8 tok/s | — | — | decode result · summary | command |
| Prefill at 4K first token 2.2 s | 2058.2 tok/s | — | — | prefill result · summary | command |
| Prefill at 8K first token 4.0 s | 2156.5 tok/s | — | — | prefill result · summary | command |
| Prefill at 16K first token 8.9 s | 1873.7 tok/s | — | — | prefill result · summary | command |
| Prefill at 32K first token 16.3 s | 2035.7 tok/s | — | — | prefill result · summary | command |
Seed 1234 applies to the two quality tests; decode, prefill and tool eval send no seed. The benchmark client ran on a separate machine that is never part of the measured group. Measured 2026-10-20. Version 2.50.0, commit 21a3511.
QA. Quality QA, 150 items, seed 1234, temperature 0, thinking off, 4 requests at once (Mia's default). An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
MMLU. Quality MMLU, 285 items (5 for each of 57 subjects, 4 answers), seed 1234, temperature 0, thinking off. An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
Decode. Decode bench, 1 request at once, 400 tokens, prompt type structured, temperature 0, thinking off. No seed is sent.
Prefill. Prefill bench at 4K, 8K, 16K and 32K tokens. sparkDash skips a size that does not fit into the context. No seed is sent.
Tool Eval Quick. Tool Eval Quick: tool-eval-bench (SeraphimSerapis, commit c61319b), 15 core scenarios (--short), --no-think, --timeout 120. Score 0 to 100. No seed is sent.
Note from the run: sparkDash 2.50.0 (@21a3511) on a client machine (never in the measured group), Quality: seed 1234, temperature 0, thinking off, a truncated answer counts as wrong; decode, prefill and tool eval send no seed; own row, never mixed with quality-v2.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 tools/quality2/sparkdash_client.py tools/quality2/cards/ds1017-exl3-eaq-2.9bpw-2x.json <box> --client <client machine> --job plan-2010 --out $RUNtool-eval-bench --base-url http://127.0.0.1:<port> --model DeepSeek-V4.1-Flash-1017 --short --no-think --timeout 120 --json-file $BENCH/sparkdash-client/runs/work/tooleval/result.json --no-live