Ash's TensorFold recipe: Google Gemma 4 31B (MLX 4-bit + DFlash drafter) on a Mac Studio

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-10. Raw files: speed result · speed contract · log.

Requests at once: the server has 4 slots (from the author's recipe). C8 exceeds the slots: the extra requests wait in the queue, and they were measured anyway. The speed test ran with Ash Hart's recipe for this model (docs/recipes/gemma-4-native.md, lines 15 and 16; release v1.2.5): tensorfold-native serve with --port 8888, --parallel 4 and --drafter. The recipe is written for the native Metal build on a Mac (its title is 'Gemma 4 31B, native Metal'); its qualification table (line 92) lists the pinned 31B 4-bit on an M5 Ultra with macOS 27, and the README of the release lists the same checkpoint for 'One Mac (M5 Ultra checked), or one GB10 on CUDA' (README, line 200). The weights are the community's MLX conversion of Google's checkpoint (mlx-community/gemma-4-31B-it-4bit, revision 696d436c; affine 4 bits, group size 64: 4 weight files, 17.18 GiB for all files of the repository, including the image tower, which is not loaded), not a quant of mine. The draft model is z-lab's DFlash for Gemma 4 31B (z-lab/gemma-4-31B-it-DFlash, revision eabd6483, 2.86 GiB), the drafter the recipe names; DFlash is a speculative decoding method that uses a lightweight block diffusion model to draft multiple tokens in parallel (z-lab's model card, line 20); the main model checks the drafted tokens. I downloaded both folders file by file with curl at those revisions (the recipe uses hf download for the same revisions) and checked the size of every file and the sha256 of the 7 large files against the Hugging Face listing; no file differed. The engine is the official release archive tensorfold-1.2.5-macos-arm64.tar.gz (tag v1.2.5, published on 8 Nov 2026; sha256 563147f1; its binary tensorfold-native, sha256 13ce9fcc), not built by me. Release 1.2.5 was the newest release when the speed and quality tests ran; release 1.2.6 was published on 10 Nov 2026 at 12:51 UTC, after them, and none of the tests on this page used it.

Differences from the author's recipe, all named: (1) A window of 262,144 tokens (--context 262144), the checkpoint's own window, instead of the default of 32,768 (recipe, lines 66 and 67). The context steps of 16K to 128K and the 32K prefill need a larger window: a prompt of 128K tokens plus its answer does not fit into 32,768 tokens. This is forced by my tests; I did not test the engine at its default window. The engine started with this window and the recipe's --parallel 4. (2) Flags I added that the recipe's command does not have: --host 127.0.0.1 (only this computer can connect), --no-update-check and --keep-warm 86400 (keeps the GPU awake between requests); I did not test whether they change the numbers. The recipe's $HOME/models folders are local folders here, and there is no --vision (the recipe's command has none; images are not tested). (3) The recipe page gives its decode numbers for an M5 Ultra Mac on Metal: plain decode at about 51 tokens a second and, with the drafter, 150 to 190 tokens a second for one stream and about 210 together for eight streams, 6 to 7 accepted tokens a round on chat and code and 2 on open-ended prose (lines 23 to 26 and 171 to 172); its receipts are greedy ones (line 168) and the measurements are the original pull request's receipts (lines 214 and 215). I measure the release with the model card's sampling (temperature 1.0, top-p 0.95, top-k 64), 512 tokens and separate code and prose prompts, so the author's figures and these are not directly comparable, and the page makes no statement about them. No run with greedy sampling is part of this page's numbers. Text only: no vision, no tool use and no agent task is measured.

The headline 76.3 tok/s is the mean of the code prompt (85.2) and the prose prompt (67.4), medians of two runs each; the spread of the two code runs is 0.9 % and of the two prose runs 2.7 % (both under the 15 % limit of our tool). Thinking was off in every request, and the tool's cache check found no cached prompt tokens in any of the 60 requests it examined. The DFlash drafts were accepted at a rate of 19.5 % of the drafted tokens (1,251 of 6,424), 2.6 tokens per step on average. Requests at once: 1, 2, 4 and 8 requests ran together. The server has 4 slots, so with 8 requests 4 of them wait in the queue; the tool measured them anyway. The total rises from 64.1 tok/s with 1 request to 116.4 with 4 and 114.5 with 8, and the speed of one request falls from 64.1 to 19.3 tok/s with 8 requests; the 8-request figures include the time spent waiting for a slot, so they are not a measure of a setup with 8 slots, and the 1-request figure is measured over the whole request, so it is below the decode speed. Long prompts: the first token of the 128K-token prompt came after 156.8 s (prefill 835.7 tok/s), see the table.

Two runs of the speed test: the first run (05:24 to 06:05 on 10 Nov 2026) threw away 12 tries and was stopped by the load guard at the second code run of the 128K step, which was still disturbed after 3 repeats; the foreign load named in its log was Apple's audio service (coreaudiod), an Apple power-statistics service (PerfPowerServices), an Apple asset service (mobileassetd) and an ffmpeg job of the dictation helper of this Mac. That run produced no value, it is not part of this page, and its raw files are not published. The dictation helper was then paused from 06:05 to 08:03 (the quality tests below ran in that window too), and the speed test ran again from the start (06:07 to 06:40); that second run is the one on this page. No foreign process was ended; the test tool holds three indexer processes (media analysis, photo analysis and Spotlight indexing) while it runs and resumes them at its end.

Load on the Mac in the second run: the controlling Claude Code session and the macOS desktop shell ran on the same Mac during every test. They, and the dictation helper, are on the guard's ignore list (named in the speed contract), so they are not counted as foreign load and stopped no step. Other Apple system services are not on that list. The load guard (a sample every half second, 3,955 samples) threw away 4 tries and repeated 3 of its 37 steps, each because one foreign process used more than 25 % of one core for at least two samples in a row: the second round of the 8-requests-at-once step (an Apple Spotlight update service, up to 99 %), the first prose run of the 32K step (Spotlight's metadata store, up to 94 %) and, twice, the first prose run of the 128K step (Apple's power-statistics service, up to 100 %, then Apple's contacts service, up to 61 %). Each repeat ran after 5 s of quiet, and the values on this page are from the repeated tries. The first-token times of the four 128K runs lie between 154.9 s and 157.8 s (the table shows the median of the four). In the other 34 steps the busiest other process stayed at or below 50.0 % of one core in a single sample. The quiet check (one sample every 10 seconds, 197 samples) saw 16 samples above 25 % of one core, up to 81.1 %: 12 of them the macOS menu bar helper (desktop shell), 1 the Claude Code session, 1 a Python process, 1 Apple's contacts service and 1 a macOS metadata worker. The quiet logs before and after the run list no foreign problems. So the Mac was not idle, and I cannot show that nothing else used it during the tests: the numbers are measured under this load. Not published: the quiet logs and the sampling file of this run (the site's filter holds back files of that kind by name), the load guard's sample file and error file, and the per-request capture folder (requests/). So the foreign-load figures above can be checked only in part: the speed contract, with the summaries of the load guard and of the quiet check and its record of each step, is published, while the guard's sample file, the quiet logs and the sampling file are not. In the published raw files the source notes (the line numbers of the author's recipe, the names of our machines and folders, a few German words) were corrected or made neutral; no measured value was touched.

python3 speed2.py --base-url http://127.0.0.1:58356/v1 --label mac256-run-gemma-s-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/mac256-run-gemma-s-speed-r0/mac256-run-gemma-s-speed-r0/speed2 --contract $RUN/mac256-run-gemma-s-speed-r0/mac256-run-gemma-s-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/models/gemma-4-31b/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 1.0, "top_p": 0.95, "top_k": 64, "min_p": 0.0} --sampling-source google/gemma-4-31B-it README 'Best Practices' (temperature 1.0, top_p 0.95, top_k 64, standardized for all use cases) = generation_config.json of mlx-community/gemma-4-31B-it-4bit @696d436c (temperature 1.0, top_k 64, top_p 0.95). No presence penalty. Seed 42 for speed/quality. --model gemma-4-31b --no-cache-prompt --server-log $LOGS/server.log --engine-version-file $LOGS/engine-version.txt --server-slots 4 --server-slots-source gemma-4-native.md:15 and :19 (--parallel 4, caps four active requests; more wait for a free slot). C8 queues 4 requests by design (more requests than slots). --server-ctx 262144 --server-ctx-source ashhart/TensorFold docs/recipes/gemma-4-native.md (tag v1.2.5): the recipe's command names no --context; lines 66 and 67 say that the context defaults to 32,768 tokens and that --context raises it to the checkpoint's 262,144. I set 262144 so that the 16K to 128K context steps fit (forced by my tests)

How well it answers

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-10.

Note from the run: MMLU-Pro (400 questions), GPQA Diamond (198), IFEval (200), LiveCodeBench (40 tasks) and RULER at 8K and 32K tokens (40 tasks each) ran with thinking off and the sampling of the model card (temperature 1.0, top-p 0.95, top-k 64), seed 42. Four requests ran at once (the server's 4 slots; each question is its own request with the same limits and seed). MMLU-Pro and GPQA Diamond ran with an answer limit of 1,024 tokens (the other pages use 48), letter only; IFEval with 4,096 tokens; LiveCodeBench with 32,768 tokens; RULER with 128 tokens. Most answers to the multiple-choice questions are a single letter (median 2 tokens). But with thinking off the model wrote a worked solution before its answer in some replies: 34 MMLU-Pro answers and 32 GPQA answers had no letter that the tool could read, and they count as wrong. The tool takes the letter when the reply starts with it or names it after a phrase such as 'answer is' or 'Answer:' (the last one wins); it does not guess. In all 34 of those MMLU-Pro replies and all 32 of those GPQA replies the last line is a letter: 27 and 18 of them end with the line 'Answer (one letter): X', 7 and 14 with the letter alone; the tool does not read such an ending and I did not score them again, so the two scores are counted by the rule and the answer files show the rest (the GPQA answers are published; the MMLU-Pro answers are held back, see below). 5 MMLU-Pro answers and 7 GPQA answers were cut off at the 1,024-token limit (counted wrong); 1 IFEval answer and 2 LiveCodeBench answers were cut off at their limits (counted wrong), and the long-text tests had none cut off: 15 cut-off answers in all. RULER by task at 8K and 32K: multi-key retrieval 10 of 10 and 10 of 10; common-word extraction 10 of 10 and 10 of 10; multi-hop question answering 9 of 10 and 5 of 10; variable tracking 10 of 10 and 7 of 10. Two of the 40 prompts at 32K were one token off the target length (32,767 and 32,769 tokens instead of 32,768; the tool counts with the model's own tokenizer). IFEval is scored with the official checker of Google Research (instruction_following_eval, commit 3ef52111), strict, at the level of the prompt. LiveCodeBench: the 40 newest tasks of the tool's list (code_generation_lite, release test6; tasks dated 2025-03-15 to 2025-04-06), pass at 1, the code runs in a container without network access; no task was lost to a sandbox error or to a missing code block. The thinking test (GPQA 30) ran with thinking on and a limit of 16,384 tokens (thinking and answer together): 2 of the 30 answers were cut off at the limit and count as wrong; the model thinks long: median 5,429 tokens and 109.7 s per question, with 4 questions running at once. These quality tests have no foreign-load guard. The Claude Code session and the macOS desktop shell ran on the same Mac, and the dictation helper was paused during these tests (see the speed notes). The sampling files (not published, see the speed notes) list the busiest other process every 10 seconds: 42 of 479 samples were above 25 % of one core (up to 703 %): 31 of them the macOS menu bar helper (desktop shell), 2 the virtual machine of the code sandbox of the LiveCodeBench test itself, 2 an Apple media conversion service, 2 a macOS metadata worker, 1 the network extension of a VPN application (191 %), 1 a file search of the controlling Claude Code session (703 %, the peak, in the thinking test) and 3 other macOS system processes. The two quality contracts record the peak of each part: 191 % in the instruction tests and 703 % in the thinking test. GPQA Diamond: the original is gated and no Hugging Face token was available, so the public mirror fingertap/GPQA-Diamond (198 rows) was used. Not published, because the site's filter held them back: the MMLU-Pro answers (mmlupro.jsonl; a model answer contains a number with a unit that the filter holds back), the IFEval answers (ifeval.jsonl; a model answer contains a word the filter reads as a machine name), the LiveCodeBench results file and generations (lcb.jsonl, lcb.gen.jsonl; model code contains a word the filter reads as a hardware setting) and the quality contract of the instruction tests (quality2-contract.json of run inst-r0; its foreign-load record names the network extension of a VPN application). The folders with the extracted programs and the sandbox runs of LiveCodeBench (lcb/code, 78 files; lcb/sandbox, 152 files) and the 80 prompt files of RULER (ruler/prompts) are not published either. The summaries and logs of all tests, the quality contract of the thinking test and of the memory test and the answers of GPQA, RULER and the thinking test are published, and the numbers of the table were counted from the complete files. These are my own tests, and no other run of this setup is mixed into the numbers on this page.

bash tools/quality2/measure_all.sh tools/quality2/cards/gemma-4-31b-mlx4bit-tf-1.2.5-official-dflash-m5ultra-1010.json --parts mmlupro --job mac256-run-gemma-q

bash tools/quality2/measure_all.sh tools/quality2/cards/gemma-4-31b-mlx4bit-tf-1.2.5-official-dflash-m5ultra-1010.json --parts gpqa --job mac256-run-gemma-q

bash tools/quality2/measure_all.sh tools/quality2/cards/gemma-4-31b-mlx4bit-tf-1.2.5-official-dflash-m5ultra-1010.json --parts ifeval --job mac256-run-gemma-q

bash tools/quality2/measure_all.sh tools/quality2/cards/gemma-4-31b-mlx4bit-tf-1.2.5-official-dflash-m5ultra-1010.json --parts lcb --job mac256-run-gemma-q

bash tools/quality2/measure_all.sh tools/quality2/cards/gemma-4-31b-mlx4bit-tf-1.2.5-official-dflash-m5ultra-1010.json --parts ruler --job mac256-run-gemma-q

bash tools/quality2/measure_all.sh tools/quality2/cards/gemma-4-31b-mlx4bit-tf-1.2.5-official-dflash-m5ultra-1010.json --parts ruler --job mac256-run-gemma-q

bash tools/quality2/measure_all.sh tools/quality2/cards/gemma-4-31b-mlx4bit-tf-1.2.5-official-dflash-m5ultra-1010.json --parts think --job mac256-run-gemma-q

How much context fits

Memory test. The context is fixed at 262,144 tokens by my start command (see the changes in the speed notes), so the server is started once with that command. Then, for each size (4K, 8K, 16K ... up to the largest size tested), one prompt that fills about 95 % of that size is sent, and the model must answer. The largest size that answered counts; it is the highest size tested, not a limit of the setup. Decode speed at that size was not measured. Peak memory is the rise of used memory over the idle value before each server start (unified memory, so GPU and CPU share it). Measured 2026-11-10. Raw files: result of every step · log · contract.

Note from the run: highest rise over idle: 87.11 GiB. The server was started once, with the context set to 262,144 tokens (the window of my start command, see the changes in the speed notes) and the recipe's --parallel 4. Then one prompt of about 95 % of each size (4K, 8K, 16K, 32K, 64K, 128K and 256K) was sent, and the model had to answer. All seven sizes were answered (finish reason stop). The largest size tested is 262,144 tokens (a prompt of 249,036 tokens), the whole window of the setup: the window of 262,144 tokens is also the checkpoint's own window, so there is no larger size to test. The memory number is the rise of the Mac's used memory over its idle value before the server was started (a Mac has one pool of memory for CPU and GPU): the Mac had 29.70 GiB in use at idle; the rise over idle was at most 52.12 GiB while the server loaded, 55.54 GiB during the 4K request, 75.06 GiB during the 128K request and 87.11 GiB during the 256K request, so the memory grows with the size of the prompt in flight. Because it is the rise of the whole Mac, other programs that used or released memory at the same time are inside the number. This engine reports no timings for this memory test, so the decode speed and the first-token columns are empty. This test ran on the evening of 10 Nov 2026 (22:38 to 22:48), after the speed and quality tests: the weights had been removed from the Mac to free disk space in between and were downloaded again with the same script (every file matches the same Hugging Face sizes and sha256 hashes), and the server was started with the same script and the same release binary. The test tool's server log of this test is not published (the site's filter holds it back: its text is German); fit.json, the test's output and its contract are published.

bash tools/quality2/measure_all.sh tools/quality2/cards/gemma-4-31b-mlx4bit-tf-1.2.5-official-dflash-m5ultra-1010.json --parts fit --job mac256-run-gemma-fit

How it plays the Gauntlet

One solo race, season seed 21, one attempt, no help.

Game rank as of Nov 10, 2026. The rank is the order of the Gauntlet finish times in this memory class. Race page and raw files.

The race ran the recipe named on its race page, with the sampling and answer limit of its lane config. The engine build of the race is not recorded in the race files.

Result (result.json) Race log (log.jsonl) Model answers (io.jsonl) System prompt Exact command Verify output Model config Race page

Also made this possible

• Engine (TensorFold 1.2.5, official release)Ash Hart@ashxhart GitHub Hugging Face Website
• Model (Gemma 4 31B, built by Google DeepMind)GoogleHugging Face Website
• MLX 4-bit weights (revision 696d436c, converted from Google's checkpoint)MLX CommunityHugging Face
• Draft model (DFlash, revision eabd6483)z-lab (Zhijian Liu)@zhijianliu_ Hugging Face GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

Mac Studio M5 Ultra, 256 GB unified memory, one box (Darwin kernel 27.0.0)

System

macOS (the Mac above)

Model

google/gemma-4-31B-it (dense 31B, 60 layers, 5 sliding : 1 global attention, vision tower not loaded), mlx-community/gemma-4-31B-it-4bit @696d436c (MLX affine 4-bit, group 64) + z-lab/gemma-4-31B-it-DFlash @eabd6483 drafter

Model (Gemma 4 31B, built by Google DeepMind)

huggingface.co/google/gemma-4-31B-it

MLX 4-bit weights (revision 696d436c, converted from Google's checkpoint)

huggingface.co/mlx-community/gemma-4-31B-it-4bit/tree/696d436c404745a59f30e4939a658162b0a9e57f

Draft model (DFlash, revision eabd6483)

huggingface.co/z-lab/gemma-4-31B-it-DFlash/tree/eabd648301ce28583cc14757912e5e0f84e152e1

Engine

TensorFold 1.2.5, the official macOS release (native Metal engine), MLX 4-bit weights with the DFlash drafter of z-lab, 262,144 tokens of context (set by me, see the changes below), up to 4 streams (--parallel 4, as in the recipe) — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.5

Recipe (docs/recipes/gemma-4-native.md)

github.com/ashhart/TensorFold/blob/v1.2.5/docs/recipes/gemma-4-native.md by Ash Hart

Steps

1. Get the recipe, the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   tensorfold-native serve \
     --host 127.0.0.1 \
     --port 8888 \
     --context 262144 \
     --parallel 4 \
     --keep-warm 86400 \
     --no-update-check \
     --drafter gemma-4-31b-dflash gemma-4-31b

   The file names are the ones we used. Run the command in the folder that holds them, or put the full path in front of each name.

   What the parts mean

   • --parallel 4how many requests it answers at the same time
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on
   • --drafter gemma-4-31b-dflashthe draft model: it guesses the next tokens and the main model checks them; without it the model decodes plain

TensorFold 1.2.5 official release archive macos-arm64 (tensorfold-native, tag v1.2.5), DFlash drafts, --context 262144, --parallel 4, prompt cache 16 GiB default, text only; tensorfold-native serve --host 127.0.0.1 --port 8888 --context 262144 --parallel 4 --keep-warm 86400 --no-update-check --drafter $HOME/models/gemma-4-31b-dflash $HOME/models/gemma-4-31b (Ash's recipe docs/recipes/gemma-4-native.md, lines 15 and 16, plus the flags and the --context named on this page; each folder holds the files of the Hugging Face repository at the revision named above)