niklaslenzBobClawblaw's TensorFold recipe: DeepSeek-V4.1-Flash-1017 (e2e3-p4-h8)
Mac Studio M5 Ultra, 256 GB unified memory, one box.
Overall score as of 25 Oct 2026. Scores are relative to the best run in the class today; a better model later moves older runs down.
How the overall score is calculated
| Part | This run | Best in class | Relative | Weight | Points |
|---|---|---|---|---|---|
| Speed, one request | decode 75.5 tok/s | best 152.5 tok/s | 49.5 % | 17.5 % | 8.7 |
| Speed, 8 at once | total 88.3 tok/s | best 112.7 tok/s | 78.4 % | 17.5 % | 13.7 |
| Quality | mean of 7 tests: 67.4 % | best mean 69.7 % | 96.8 % | 25 % | 24.2 |
| Browser check | 8.5 of 10 | 85.0 % | 5 % | 4.3 | |
| My rating | 3 of 10 | 30.0 % | 35 % | 10.5 | |
| Overall | 100 % | 61.3 |
The seven quality tests: MMLU-Pro 67.5 · GPQA 45.0 · IFEval 90.5 · LiveCodeBench 60.0 · long text 8K 70.0 · long text 32K 62.5 · GPQA 30 with thinking 76.7.
The speed part is half the decode speed with one request and half the total speed with 8 requests at once, each relative to the best run of this class; a run without a total for 8 requests at once gets the better of its decode speed alone and the mix with its highest measured total. Quality is relative to the best run of this class, the browser check and my rating are out of 10. The parts are rounded one by one. sparkDash, the memory test and speculative-decoding numbers are not part of it. How the score is made, and the assumption behind it.
What it builds as an agent
Task: Voxel world (3D scene, one HTML file). One attempt, no help. Counted: the final file counts.
The exact promptcopy it and try it yourself↓
This is the task prompt word for word, as we handed it to the agent harness. The harness adds its own system prompt and tools, and some harnesses add short context notes of their own. A run that goes past 90 minutes also gets a short “finish now” message, as the prompt says. To try it yourself, give it to your own coding agent: it needs a shell and a working folder.
Design and create the most impressive, creative, elaborate and detailed
voxel art world you are capable of. The theme and style are ENTIRELY your
choice — pick whatever world you can execute best and push it as far as
you can: scale, variety, detail, color, animation, atmosphere, life.
Surprise me.
Hard requirements:
- Everything ends up in ONE single self-contained HTML file (game.html in
the current working directory) that opens and runs directly in Chrome.
Use WebGL and whatever libraries you like, but embed everything —
no internet access at view time, no external files.
- The scene must present itself: include an automatic slow cinematic
camera tour (orbiting the world, occasionally zooming in on highlights)
that starts on load, plus mouse controls (orbit/zoom) for manual viewing.
- All user-facing text (UI, HUD, titles, descriptions, hints) must be in
ENGLISH, regardless of the language of any project files you encounter.
When finished, verify your HTML actually loads and renders without
console errors, then state DONE with the absolute path of game.html.
Do not ask questions.
How you work (same for every model):
- You work alone in the folder /work (this is the current working directory). Nobody will answer questions. Aim to finish in about 60 minutes. After 90 minutes you will get a message to finish; then you have at most 30 more minutes. Hard stop after 120 minutes.
- Tools on this machine: shell, files, python3 with numpy and Pillow, ffmpeg, node, and `shot.sh game.html` (opens the page in a headless browser, saves a screenshot and prints console errors). There is no internet and no access to other machines.
- Keep a working /work/game.html at all times. If you are stopped, the last version that loaded without errors counts.
- JavaScript libraries available offline in the test browser, through an import map with these exact URLs: "three": "https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.module.js", "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/", "simplex-noise": "https://cdn.jsdelivr.net/npm/simplex-noise@4.0.3/dist/esm/simplex-noise.js", "@tweenjs/tween.js": "https://cdn.jsdelivr.net/npm/@tweenjs/tween.js@25.0.0/dist/tween.esm.js", "cannon-es": "https://cdn.jsdelivr.net/npm/cannon-es@0.20.0/dist/cannon-es.js". No other external resources.
- After the scene is built, set `window.scene` to the Three.js scene (if you use Three.js) and `window.ready = true`.
Play it yourself
Open full screenAI-generated The page this setup built, running live in your browser: click into it and use the controls it shows itself (usually drag to look around, scroll to zoom). It needs WebGL and can be slow on a weak graphics card.
The browser check
- page loads without page error1 / 1
- window.ready === true within 30 s1 / 1
- no console errors (whole session)1 / 1
- frame is not blank1 / 1
- frame rate in software GL (rAF over 5 s)0 / 1
- world: scene size 0.5 (parts >= 80 or >= 20000 triangles; canvas fallback), visible text English if any 0.51 / 1
- automatic camera tour: the view changes on its own in two successive ~5 s windows (1 each)2 / 2
- animation in the world (nodes / instances / particles moving besides the camera)1 / 1
- OrbitControls: drag and wheel change the picture beyond the animation baseline0.5 / 1
Measured 2026-10-25. Raw files: run info · command · final choice · contract · sampling · gate stats · gate log · check · screenshot · clip.mp4 · web.mp4 · web-poster.jpg.
Other processes used up to 400 % CPU during the run. Browser check 8.5 out of 10 (voxel-world task), measured on a second machine: the check that ran together with the agent run did not start, because the Docker VM on the Mac has 4 CPU cores and the check needs 16. I re-ran the same check (same image q2-harness:1, same tool files, sha256 checked) on box-2, a Linux box with 20 cores, from a copy of the run's final index.html (same sha256) and took over its score; its check.json is the evidence (folder recheck-box-2). The page loads without a page error, is ready after 0.15 s, has no console errors, is not blank, shows a world (48,435 parts, 587,520 triangles, English text), flies the automatic camera tour in both windows and animates (14 animated nodes). The 1.5 lost points are the frame rate (1.55 frames per second in software rendering without a GPU, 10 give half a point and 20 the full point) and half of the orbit-controls test (the mouse wheel changed the picture, the drag changed it less than the check needs). The agent (DeepSeek Harness 0.2.1-rc.2, the harness DeepSeek ships) took 45.7 minutes, one try, no help; the “finish now” message (from 90 minutes) was not needed. 63 requests, no error, no request blocked (5,087,553 prompt tokens, 112,065 answer tokens, largest prompt 141,044 of 524,288 tokens). 27 versions of the page were seen on the way, 23 loaded without errors, and the last one counts; the first version came after 8.4 minutes and the first one that loaded without errors after 11.6 minutes. The limit per answer was 262,144 tokens, the value on the DeepSeek model card (the author's recipe sets less, see the context note). Thinking was on, sampling temperature 0 as the author runs it. What it built: AETHERIA, a voxel world of six floating islands with a citadel under a floating sun crystal, a river that falls into the clouds, a crystal hollow, terraced gardens, a beacon tower, an airship, a sky whale, birds and clouds, and a camera tour of 8 chapters. At the end of the run the model's thinking repeated one line (“We use window.ready true. Good.”) 44 times before it answered DONE; the page was finished by then (the last clean version came after 43.5 minutes).
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
run_showcase.sh --harness dsh-021 --task VX --upstream http://127.0.0.1:54681/v1 --model DeepSeek-V4.1-Flash --sampling-file $RUN/sampling-show.json --out $RUN/raw/quality-v2-261025-mac-ds41-bob-tf-2510-show-r8/show --limit-s 7200 --inject-after-s 5400 --poll-s 20 --ctx 524288It also built this
Task: Airship dock (3D scene, one HTML file). One attempt, no help. Counted: the final file counts. This extra build test is shown for the full picture; it is not part of the ranking.
The exact promptcopy it and try it yourself↓
This is the task prompt word for word, as we handed it to the agent harness. The harness adds its own system prompt and tools, and some harnesses add short context notes of their own. A run that goes past 90 minutes also gets a short “finish now” message, as the prompt says. To try it yourself, give it to your own coding agent: it needs a shell and a working folder.
Design and build the most impressive, creative, elaborate and detailed steampunk airship dock voxel world you are capable of. Theme: a steampunk airship dock. Everything else (style, story, extra details) is your choice.
How you work:
- You work alone in the folder /work. Nobody will answer questions. Aim to finish in about 60 minutes. After 90 minutes you will get a message to finish; then you have at most 30 more minutes. Hard stop after 120 minutes. Spend the time where it shows in the result.
- Tools on this machine: shell, files, python3 with numpy and Pillow, ffmpeg, node, and `shot.sh index.html` (opens the page in a headless browser, saves a screenshot and prints console errors). There is no internet and no access to other machines.
- You may create your own assets locally with these tools (textures, sky, sounds, music) and embed them in the file as data URIs. Do not download anything.
- Keep a working /work/index.html at all times: write a first working version early, then improve it step by step. If you are stopped, the last version that loaded without errors counts.
- Check your work with `shot.sh` as often as you like, fix every error, and keep improving until you are satisfied. Then stop.
Deliverable: ONE self-contained file /work/index.html that opens and runs in Chrome.
- JavaScript libraries: only these, through an import map with these exact URLs (they work offline in the test browser). Use whatever helps, including all Three.js addons (post-processing, sky, water, noise and so on):
- "three": "https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.module.js"
- "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/"
- "simplex-noise": "https://cdn.jsdelivr.net/npm/simplex-noise@4.0.3/dist/esm/simplex-noise.js"
- "@tweenjs/tween.js": "https://cdn.jsdelivr.net/npm/@tweenjs/tween.js@25.0.0/dist/tween.esm.js"
- "cannon-es": "https://cdn.jsdelivr.net/npm/cannon-es@0.20.0/dist/cannon-es.js"
- No other external resources. Build every object from cubes (voxels); InstancedMesh is allowed.
- The page fills the whole window and resizes with it. All visible text is in English.
- After the scene is built, set `window.scene` to the Three.js scene and `window.ready = true`.
- An automatic slow cinematic camera tour starts on load. Mouse input switches to OrbitControls (drag to rotate, scroll to zoom).
- Sound, if any, starts only after the first click (browsers block autoplay).
- A small label in the top-left corner: "Steampunk Airship Dock".
Minimum content (go far beyond it):
1. A tall iron docking tower with platforms, stairs or ladders, and a mooring arm at the top.
2. At least two airships, each with a balloon envelope, a gondola and propellers.
3. Big brass gears on the tower or the buildings.
4. Pipes and at least three chimneys that emit rising steam particles.
5. At least one crane and a stack of cargo crates.
6. A ground area with a workshop building and a small rail track.
7. Propellers and gears rotate, steam rises and fades, and one airship slowly approaches the tower, docks, waits and leaves again, in a loop.
8. Warm sunset lighting with shadows.
Play it yourself
Open full screenAI-generated The page this setup built, running live in your browser: click into it and use the controls it shows itself (usually drag to look around, scroll to zoom). It needs WebGL and can be slow on a weak graphics card.
Measured 2026-10-25. Raw files: run info · command · final choice · contract · sampling · gate stats · check · screenshot · clip.mp4.
Other processes used up to 399 % CPU during the run. Extra build, not part of the score. This run used the airship-dock prompt (the one the 16 GB and 32 GB classes use), not the voxel-world prompt this page is ranked on. Same Mac, same server settings, same agent and same limits as the voxel-world run above; only the prompt differs. I rated it 4.5 out of 10; it is not part of the rating of this page. Browser check 7.5 out of 10, measured on box-2 from the run's final file (the check on the Mac did not start, see above): the frame rate lost its point (3.43 frames per second in software rendering), the check for the required objects gave 1.5 of 2 (no airship found by name or by structure), and the orbit-controls check gave 0 of 1. The agent needed 93.1 minutes, one try, no help; the “finish now” message came at 90.3 minutes and was sent with 5 requests. 19 versions of the page were seen on the way, all 19 loaded without errors, and the last one counts. 289 requests, no error, none blocked (39,163,541 prompt tokens, 367,809 answer tokens, largest prompt 251,828 of 524,288 tokens). Answer limit 262,144 tokens, context 524,288 tokens, thinking on, temperature 0.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
run_showcase.sh --harness dsh-021 --task K1 --upstream http://127.0.0.1:54783/v1 --model DeepSeek-V4.1-Flash --sampling-file $RUN/sampling-show.json --out $RUN/raw/quality-v2-261025-mac-ds41-bob-tf-2510-show-r1/show --limit-s 7200 --inject-after-s 5400 --poll-s 20 --ctx 524288How fast it runs
| Measurement | Result |
|---|---|
| Decode | 75.5 tok/s |
| Words per second | 54.0 words/s |
| Characters per second | 328.8 chars/s |
| Decode, code prompt | 80.5 tok/s |
| Decode, prose prompt | 70.5 tok/s |
| Spread of the decode runs (CV) | 0.9 % |
| Spread of the code runs (CV) | 0.8 % |
| Spread of the prose runs (CV) | 0.9 % |
| Decode, window 1 | 81.1 tok/s |
| Prefill at 2K | 1536.5 tok/s |
| Prefill at 8K | 1742.9 tok/s |
| Prefill at 32K | 1766.3 tok/s |
| Time to first token at 2K | 1.3 s |
| Time to first token at 8K | 4.7 s |
| Time to first token at 32K | 18.6 s |
| Draft acceptance rate | 56.3 % |
| Mean accepted length | 2.0 tokens |
| 1 request at once (total) | 62.5 tok/s |
| 2 at once (total) | 72.1 tok/s |
| 4 at once (total) | 88.2 tok/s |
| 8 at once (total) | 88.3 tok/s |
| 1 request at once (per request) | 62.5 tok/s |
| 2 at once (per request) | 38.4 tok/s |
| 4 at once (per request) | 22.7 tok/s |
| 8 at once (per request) | 15.6 tok/s |
| Server slots (from the recipe) | 4 slots |
| Long answer, wall time | 32.9 s |
Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-10-25. Raw files: speed result · speed contract · log · long answer.
Requests at once: the server has 4 slots (from the author's recipe). C8 exceeds the slots: the extra requests wait in the queue, and they were measured anyway. Other processes used up to 111 % CPU during the run. The speed test ran with the author's setup: BobClawblaw's recipe (BobClawblaw/deepseek-v41-flash-mlx-m5-ultra, commit 2d47ae05, 23 Oct 2026), started with his own tool (python3 -m mac up) and the settings of his recipe.toml lines 31 to 39 unchanged: context 524,288 tokens shared by the streams, 4 streams (the most this server allows, mac/config.py: parallel must be 1 to 4), DSpark drafts on, loop guard on, prompt cache 6 GiB, temperature 0 (greedy, as the author runs it). Engine: BobClawblaw's TensorFold branch deepseek-v41-main at commit 72a552ed (his fork of Ash Hart's TensorFold, Metal, built here from the source with Zig 0.17.0). Weights: BobClawblaw/DeepSeek-V4.1-Flash-1017-TensorFold-e2e3-p4-h8 at revision 9012d306 (all 52 big files checked against the Hugging Face hashes). My startup script only changes the port and the local paths, and, while the Docker VM for the browser tests runs, the startup memory check (2 GiB margin instead of 6); that is no server setting. The DeepSeek model card suggests temperature 1.0; I kept the author's temperature 0. Thinking is on by default in the author's server; the speed test sends the request flag for no thinking, seed 42, no prompt cache. The headline 75.53 tok/s is the mean of the code prompt (80.54) and the prose prompt (70.52), medians of two runs each (spread 0.8 % and 0.9 %). The drafts were accepted at a rate of 56.3 % (2.04 tokens per round). Requests at once: the server runs at most 4 streams, so C8 sends 8 requests at once and 4 of them wait for a free stream (two waves); that is why 8 at once is no faster than 4 at once. A website build (low priority, nice 19) ran on this Mac from 05:43 to 06:05 while this card was being measured. It overlapped the last 5.6 minutes of the quality block, which was the long-text test (the load log of that block shows the build's Python process at about 100 % of one core from 05:43:29 to 05:49:05); accuracy does not depend on speed, and the test took its answers from the server as usual. It also overlapped an attempt of the thinking test that was cancelled and thrown away; the thinking test that counts (06:42 to 07:10) ran alone. The speed test ran alone: the last build before it ended at 02:43, the speed test ran from 02:44, and the highest foreign load in the load log (a sample every 10 seconds) was 110.6 % of one core, from the macOS window server. The voxel-world build (04:05 to 04:51) ran before the website build; the memory test (08:11 to 08:18) ran after it. The Mac's own background services (media analysis, Spotlight, helper processes) show up in the load log all night, and the quiet check before every part passed.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 speed2.py --base-url http://127.0.0.1:62902/v1 --label quality-v2-261025-mac-ds41-bob-tf-2510-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/quality-v2-261025-mac-ds41-bob-tf-2510-speed-r0/quality-v2-261025-mac-ds41-bob-tf-2510-speed-r0/speed2 --contract $RUN/raw/quality-v2-261025-mac-ds41-bob-tf-2510-speed-r0/quality-v2-261025-mac-ds41-bob-tf-2510-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1 --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --tokenizer-json $HOME/models/bob-dsv41-tf-metal-9012d306/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.0, "top_p": 1.0} --sampling-source The author runs greedy: recipe.toml:37 temperature = "0"; README 'Quality' (recipe-harness, greedy) and 'Speed' (greedy). DeepSeek model card (deepseek-ai rev 1bae0e10) recommends temperature 1.0 / top_p 1.0: NOT used, SENDEPLAN 8 (settings as the author runs them). Open question: the author's greedy setting or the card's 1.0 (README 'Known limits': the author's own thinking-on greedy GPQA loops, loop_guard closes them). --model DeepSeek-V4.1-Flash --no-cache-prompt --server-log $RUN/logs/server-A.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 4 --server-slots-source BobClawblaw/deepseek-v41-flash-mlx-m5-ultra @2d47ae05 recipe.toml:38 parallel = 4 (4 is the maximum: mac/config.py validate 'parallel must be 1 to 4'; the streams share context 524288, so the full context is kept at 4); README 'Settings'. C8 therefore measures the queue (2 waves) by design.How well it answers
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| MMLU-Pro 400 fixed questions, letter only | 67.5 % ± 4.6 pts | 270 / 400 | 1 | answers · summary · log · contract | command |
| GPQA Diamond 198 questions, letter only | 45.0 % ± 6.9 pts | 89 / 198 | 9 | answers · summary · log · contract | command |
| IFEval 200 fixed prompts, strict | 90.5 % ± 4.1 pts | 181 / 200 | 0 | answers · summary · log · contract | command |
| LiveCodeBench 40 newest tasks, the code runs | 60.0 % ± 14.5 pts | 24 / 40 | 14 | summary · log · contract | command |
| Long text 8K 4 tasks x 10 at 8192 tokens | 70.0 % ± 13.7 pts | 28 / 40 | 0 | log · contract · answers · summary | command |
| Long text 32K 4 tasks x 10 at 32768 tokens | 62.5 % ± 14.4 pts | 25 / 40 | 0 | log · contract · answers · summary | command |
| GPQA 30, thinking on 30 questions, limit 16,384 tokens | 76.7 % ± 14.6 pts median 2,098 tokens, 74.7 s per question | 23 / 30 | 0 | log · contract · answers · summary | command |
Thinking off, except the last test. Seed 42. Sampling as the author runs it, see the note below. A cut-off answer counts as wrong. Measured 2026-10-25.
Note from the run: Quality ran on this card in several blocks on 25 Oct 2026 (the long-text and the multiple-choice tests together, LiveCodeBench alone, the thinking test alone). The questions ran 4 at a time, the server's stream count (same questions, limits and seed 42 for every question); the median time per question of the thinking test (74.7 s) was measured under that load. MMLU-Pro and GPQA Diamond ran with an answer limit of 1,024 tokens (the model often writes a short explanation before the letter), IFEval with 4,096, LiveCodeBench with 32,768, the long-text tests with 128 tokens. Answers cut off at their limit count as wrong: 14 of the 40 LiveCodeBench answers (limit 32,768 tokens; these answers end in the middle of reasoning that the model wrote into the reply even with thinking off), 9 of the 198 GPQA Diamond answers (limit 1,024 tokens, 5 more had no answer letter) and 1 of the 400 MMLU-Pro answers (limit 1,024 tokens). No other test lost an answer. GPQA Diamond ran from the public mirror fingertap/GPQA-Diamond (198 questions), because the original is gated and I had no access token. Sampling: temperature 0 and top_p 1.0 in every test, as the author's server and harness run it (recipe.toml:37, README 'Quality'); the DeepSeek card suggests 1.0. Thinking was off in all tests except GPQA with thinking (30 questions, limit 16,384 tokens, none cut off). These are my own tests; no other run is mixed into the quality numbers on this page. A website build (low priority, nice 19) ran on this Mac from 05:43 to 06:05 while this card was being measured. It overlapped the last 5.6 minutes of the quality block, which was the long-text test (the load log of that block shows the build's Python process at about 100 % of one core from 05:43:29 to 05:49:05); accuracy does not depend on speed, and the test took its answers from the server as usual. It also overlapped an attempt of the thinking test that was cancelled and thrown away; the thinking test that counts (06:42 to 07:10) ran alone. The speed test ran alone: the last build before it ended at 02:43, the speed test ran from 02:44, and the highest foreign load in the load log (a sample every 10 seconds) was 110.6 % of one core, from the macOS window server. The voxel-world build (04:05 to 04:51) ran before the website build; the memory test (08:11 to 08:18) ran after it. The Mac's own background services (media analysis, Spotlight, helper processes) show up in the load log all night, and the quiet check before every part passed.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/ds41-1017-bob-tf-e2e3-p4-h8-tfmetal-72a552ed-2510.json --parts mmlupro --job quality-v2-261025-mac-ds41-bob-tf-2510bash tools/quality2/measure_all.sh tools/quality2/cards/ds41-1017-bob-tf-e2e3-p4-h8-tfmetal-72a552ed-2510.json --parts gpqa --job quality-v2-261025-mac-ds41-bob-tf-2510bash tools/quality2/measure_all.sh tools/quality2/cards/ds41-1017-bob-tf-e2e3-p4-h8-tfmetal-72a552ed-2510.json --parts ifeval --job quality-v2-261025-mac-ds41-bob-tf-2510bash tools/quality2/measure_all.sh tools/quality2/cards/ds41-1017-bob-tf-e2e3-p4-h8-tfmetal-72a552ed-2510.json --parts lcb --job quality-v2-261025-mac-ds41-bob-tf-2510bash tools/quality2/measure_all.sh tools/quality2/cards/ds41-1017-bob-tf-e2e3-p4-h8-tfmetal-72a552ed-2510.json --parts ruler --job quality-v2-261025-mac-ds41-bob-tf-2510bash tools/quality2/measure_all.sh tools/quality2/cards/ds41-1017-bob-tf-e2e3-p4-h8-tfmetal-72a552ed-2510.json --parts ruler --job quality-v2-261025-mac-ds41-bob-tf-2510bash tools/quality2/measure_all.sh tools/quality2/cards/ds41-1017-bob-tf-e2e3-p4-h8-tfmetal-72a552ed-2510.json --parts think --job quality-v2-261025-mac-ds41-bob-tf-2510How much context fits
| Context size | Prompttokens | Filled | First token | Prefilltok/s, as the server reports it | Decodetok/s | Peak memoryGiB | Result |
|---|---|---|---|---|---|---|---|
| 4K 4,096 tokens | 3,891 | 95.0 % | — | — | — | 203.64 | answered (length) |
| 8K 8,192 tokens | 7,782 | 95.0 % | — | — | — | 205.90 | answered (length) |
| 16K 16,384 tokens | 15,564 | 95.0 % | — | — | — | 206.87 | answered (stop) |
| 32K 32,768 tokens | 31,130 | 95.0 % | — | — | — | 205.60 | answered (stop) |
| 64K 65,536 tokens | 62,259 | 95.0 % | — | — | — | 207.62 | answered (stop) |
| 128K 131,072 tokens | 124,518 | 95.0 % | — | — | — | 206.35 | answered (stop) |
Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed is measured on that answer at the largest size. Peak memory is the rise of used memory over the idle value before each server start (unified memory, so GPU and CPU share it). Measured 2026-10-25. Raw files: result of every step · log · contract.
Note from the run: The idle memory before the steps was not constant: 31.0 GiB before the first step, 31.3 GiB before the last one. Each peak is counted over the idle value before its own step. The memory test goes up to 131,072 tokens of context (the steps of the test are 4K, 8K, 16K, 32K, 64K and 128K); the author's own server setting is 524,288 tokens, which I did not test as a memory step. For each size the server was started again with that context and the author's 4 streams, one prompt that fills about 95 % of it was sent (prompts of 3,891 to 124,518 tokens), and the model answered at every size. The memory number is the rise of used memory over the idle value of the Mac (unified memory): between 203.64 and 207.62 GiB at every step, 206.35 GiB at 131,072 tokens (the highest single step was 207.62 GiB at 65,536 tokens). The weights are held in memory (about 200 GiB); the Engram tables (189 GiB) are read from the SSD. The decode speed at the largest size was not recorded.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/ds41-1017-bob-tf-e2e3-p4-h8-tfmetal-72a552ed-2510.json --parts fit --job quality-v2-261025-mac-ds41-bob-tf-2510Also made this possible
- Model
DeepSeekHugging Face @deepseek_ai GitHub Website - Engine (TensorFold)
Ash Hart@ashxhart GitHub Hugging Face Website
Run it yourself
What you need and how to start it, step by step. The results below come from exactly this setup.
What you need
- Hardware
- Mac Studio M5 Ultra, 256 GB unified memory, one box.
- System
- macOS (the Mac above)
- Model
- DeepSeek-V4.1-Flash-1017 (BobClawblaw/DeepSeek-V4.1-Flash-1017-TensorFold-e2e3-p4-h8 @9012d306)
- Engine
- BobClawblaw's TensorFold branch deepseek-v41-main (Metal) on one Mac Studio M5 Ultra, 2/3-bit MLX weights with DSpark drafts, 524,288 tokens of context, 4 streams — the program that loads the model and answers requests · github.com/ashhart/TensorFoldBuild and settings we used: TensorFold native Metal engine, deepseek-v41-main @72a552ed (BobClawblaw fork, Zig 0.17.0 source build Oct 25, 2026)
- Recipe, TensorFold branch (deepseek-v41-main) and quantised model files
- github.com/BobClawblaw/deepseek-v41-flash-mlx-m5-ultra by BobClawblaw
Steps
Get the recipe and the engine from the list above. The list has no direct download link for the model file itself. The base model is huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017.
Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.
tensorfold-native serve \ --host 127.0.0.1 \ --port 8888 \ --name DeepSeek-V4.1-Flash \ --context 524288 \ --max-tokens 32768 \ --temperature 0 \ --parallel 4 \ --keep-warm 86400 \ --no-update-check \ --thinking \ --thinking-reserve 49152 \ --loop-guard \ --prompt-cache-gib 6 \ --prompt-cache-over-cap \ --vision \ --vision-image-tokens 4096 \ --vision-max-images 4 bob-dsv41-tf-metal-9012d306The file names are the ones we used. Run the command in the folder that holds them, or put the full path in front of each name.
What the parts mean
--parallel 4how many requests it answers at the same time--host 127.0.0.1who can connect: only this computer--port 8888the network port the server listens on
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the start text exactly as our run recorded it, with the engine build in front, so anyone can repeat the result.
TensorFold native Metal engine, deepseek-v41-main @72a552ed (BobClawblaw fork, Zig 0.17.0 source build Oct 25, 2026); tensorfold-native serve --host 127.0.0.1 --port 8888 --name DeepSeek-V4.1-Flash --context 524288 --max-tokens 32768 --temperature 0 --parallel 4 --keep-warm 86400 --no-update-check --thinking --thinking-reserve 49152 --loop-guard --prompt-cache-gib 6 --prompt-cache-over-cap --vision --vision-image-tokens 4096 --vision-max-images 4 $HOME/models/bob-dsv41-tf-metal-9012d306 (via python3 -m mac up, $HOME/queue-2410-mac/tf-serve.sh start; recipe.toml:31-39)Context and sampling: 524,288 tokens shared by 4 streams (the author's default, recipe.toml:31; the model window is 1,048,576), 4 streams (recipe.toml:38, the most the server takes), DSpark drafts on (recipe.toml:32), loop guard on, thinking reserve 49,152, prompt cache 6 GiB, temperature 0 / greedy as the author's server and harness run it (recipe.toml:37, README 'Quality'); seed 42; no prompt cache requested by the tests. Thinking: off per request in speed and the instruct tests, on for GPQA with thinking and the build test. The answer limit for the build test was 262,144 tokens, the value the DeepSeek model card names (at least 256K); the author's recipe sets 32,768 plus a thinking reserve of 49,152, which is lower, so I raised it for the test. Agent for the build test: DeepSeek Harness 0.2.1-rc.2, the harness DeepSeek ships (the same for every DeepSeek run). Hardware: Mac Studio M5 Ultra, 256 GB unified memory, one box.. All raw files of this setup: raw/v2/deepseek-v41-flash-mac-tensorfold-metal-bob.
Mia's own benchmark, run unchanged
This is Mia's own benchmark tool (commit 21a3511), run unchanged. It has its own section on this page, and it is never mixed with my numbers. A cut-off answer counts as wrong here, so a value can differ from sparkDash's own screen.
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| Quality QA 150 questions | 91.3 % ± 4.6 pts | 137 / 150 | 0 | summary | command |
| Quality MMLU 285 questions | 92.6 % ± 3.1 pts | 264 / 285 | 0 | summary | command |
| Tool Eval Quick 15 scenarios, tool use | 83 / 100 | 12 / 15 passed | — | tool eval result · tool eval report · tool eval command · summary | command |
| Decode, 1 request at once first token 125 ms | 145.1 tok/s | — | — | decode result · summary | command |
| Prefill at 4K first token 2.4 s | 1882.9 tok/s | — | — | prefill result · summary | command |
| Prefill at 8K first token 4.4 s | 1931.1 tok/s | — | — | prefill result · summary | command |
| Prefill at 16K first token 8.8 s | 1895.5 tok/s | — | — | prefill result · summary | command |
| Prefill at 32K first token 18.2 s | 1816.9 tok/s | — | — | prefill result · summary | command |
Seed 1234 applies to the two quality tests; decode, prefill and tool eval send no seed. The benchmark client ran on a separate machine that is never part of the measured group. Measured 2026-10-26. Version 2.50.0, commit 21a3511.
QA. Quality QA, 150 items, seed 1234, temperature 0, thinking off, 4 requests at once (Mia's default). An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
MMLU. Quality MMLU, 285 items (5 for each of 57 subjects, 4 answers), seed 1234, temperature 0, thinking off. An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
Decode. Decode bench, 1 request at once, 400 tokens, prompt type structured, temperature 0, thinking off. No seed is sent.
Prefill. Prefill bench at 4K, 8K, 16K and 32K tokens. sparkDash skips a size that does not fit into the context. No seed is sent.
Tool Eval Quick. Tool Eval Quick: tool-eval-bench (SeraphimSerapis, commit c61319b), 15 core scenarios (--short), --no-think, --timeout 120. Score 0 to 100. No seed is sent.
Note from the run: sparkDash runs its own tests (its own QA and MMLU questions at temperature 0, thinking off, a cut-off answer counts as wrong), so its decode and prefill values are not comparable with Speed v2 above: its decode test uses a structured prompt, Speed v2 a code and a prose prompt. The sparkDash client ran on another machine (a laptop) and was never part of the measured group.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 tools/quality2/sparkdash_client.py tools/quality2/cards/ds41-1017-bob-tf-e2e3-p4-h8-tfmetal-72a552ed-2510.json <box> --client <client machine> --job quality-v2-261025-mac-ds41-bob-tf-2510 --out $RUNtool-eval-bench --base-url http://127.0.0.1:<port> --model DeepSeek-V4.1-Flash --short --no-think --timeout 120 --json-file $BENCH/sparkdash-client/runs/quality-v2-261025-mac-ds41-bob-tf-2510-sparkdash-r13/tooleval/result.json --no-live