Ash's TensorFold recipe: NVIDIA Nemotron 3.5 Lightning 30B-A3B (MLX 4-bit + MTP drafts) on a Mac Studio

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-10. Raw files: speed result · speed contract · log.

Requests at once: the server has 8 slots (the engine's default, up to 8 streams; the recipe sets no value). The speed test ran with Ash Hart's recipe for this model (docs/recipes/nemotron-3.5.md, section Run, lines 16 and 17; main as I read it on 10 Nov 2026): tensorfold serve TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit --name bench. The weights are the TensorFold team's own MLX conversion of NVIDIA's BF16 checkpoint (affine 4 bits, group size 64: 4 shards and the MTP head file mtp-4bit.safetensors, about 17.3 GiB, revision d9d758fb), not a quant of mine. I started the engine from a folder with those files; some of them were copied from another machine of mine because the direct download from Hugging Face ran far too slowly. All 14 files match the sizes in the Hugging Face file listing and the 6 large files (4 shards, the MTP head and the tokenizer) match its sha256 hashes. The engine is the official release archive tensorfold-1.2.5-macos-arm64 (tag v1.2.5, sha256 563147f1; its binary tensorfold-native, sha256 13ce9fcc), not built by me.

Differences from the author's recipe, all named: (1) A window of 262,144 tokens (--context 262144), the window of the checkpoint (its config.json), instead of the 16,384 tokens that the recipe's CUDA section names as the default request (line 82) or the 32,768 of its own measurements (line 157). The context steps of 16K to 128K and the 32K prefill need a larger window: a prompt of 128K tokens plus its answer does not fit into 16,384 or 32,768 tokens. This is forced by my tests; I did not test the engine's own default for this model. (2) The streams: the recipe sets no --parallel, so the engine's default --parallel auto applies, which asks for up to 8 streams as memory and the engine allow (README, lines 104 and 343). The README tells you to read the accepted count at startup, but the start line of this release does not print it, so I cannot quote the number from the log. What I did check: in the two rounds with 8 requests at once the engine's own queue field in its log (the log is not published) shows that no request waited longer than 1.231 s; that fits prompts processed one after the other, not a wait for a free stream. So the page counts 8 slots. The label 'recipe' for the slot count in the raw files is the test tool's name for a value taken from the setup, not a setting of the recipe. (3) Flags I added that do not change the numbers: --host 127.0.0.1, --port 8888, --no-update-check, --keep-warm 86400 (keeps the GPU awake between requests) and no --name. (4) The recipe page measures with greedy decoding (temperature 0) and short replies of 96 to 256 tokens (lines 157 and 211); I measure with the model card's sampling, 512 tokens and separate code and prose prompts, and I use the official release, not necessarily the engine state of the author's own measurements. So the author's figures and these are not directly comparable, and the page makes no statement about them. Text only: no vision, no tool use and no agent task is measured.

The headline 306.9 tok/s is the mean of the code prompt (302.7) and the prose prompt (311.1), medians of two runs each (spread 1.7 % and 5.4 %). Thinking was off in every request, and the tool's cache check found no cached prompt tokens in any of the 60 requests it examined. The drafts were accepted at a rate of 48.4 % of the drafted tokens (1,140 of 2,356), 2.3 tokens per step on average. Requests at once: 1, 2, 4 and 8 requests ran together. The total rises from 272.5 tok/s with 1 request to 533.5 tok/s with 8; the speed of one request falls from 272.5 to 70.6 tok/s (the 1-request figure includes the prompt time, so it is below the decode speed).

Load on the Mac: the controlling Claude Code session and the macOS desktop shell ran on the same Mac during every test. They, and a dictation helper, are on the guard's ignore list (named in the speed contract), so they are not counted as foreign load and stopped no step. Other Apple system services are not on that list. The load guard (a sample every half second, 533 samples) threw away and repeated 2 of its 37 steps, each because Apple's audio service used more than 25 % of one core for two samples in a row: the first run of the 32K prefill (up to 49 %) and the second code run of the 16K context step (up to 63 %). The repeats ran once the Mac was quiet again; the discarded tries are listed in the raw files. The quiet check (one sample every 10 seconds) saw at most 117 % of one core, from the macOS menu bar helper. So the Mac was not idle, and I cannot show that nothing else used it during the tests: the numbers are measured under this load.

python3 speed2.py --base-url http://127.0.0.1:64872/v1 --label mac256-run-nemo-s-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/mac256-run-nemo-s-speed-r0/mac256-run-nemo-s-speed-r0/speed2 --contract $RUN/mac256-run-nemo-s-speed-r0/mac256-run-nemo-s-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/models/nemotron-lightning/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 1.0, "top_p": 0.95} --sampling-source TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit @d9d758fb README (mlx_lm and oMLX examples: temperature 1.0, top_p 0.95) = generation_config.json (temperature 1.0, top_p 0.95, do_sample true; no top_k named). No top_k, min_p or penalties are named by the maker or the recipe: server defaults. Seed 42 for speed/quality. Reasoning is on in the chat template by default; the harness switches it per request (chat_template_kwargs.enable_thinking). --model nemotron-3.5-lightning --no-cache-prompt --server-log $LOGS/server.log --engine-version-file $LOGS/engine-version.txt --server-slots 8 --server-slots-source README.md (TensorFold, main): '--parallel N|auto ... Default auto asks for up to 8 as memory and the engine allow'; the Lightning recipe (nemotron-3.5.md Run) sets no --parallel, so the default is used; the accepted count is read from the server start line (see log). C8 fills 8 slots if accepted. --server-ctx 262144 --server-ctx-source ashhart/TensorFold docs/recipes/nemotron-3.5.md (the recipe's command sets no context; line 82: the default requested context is 16,384 tokens; line 157: the author's own measurement uses --context 32768) and the checkpoint's own window (config.json max_position_embeddings 262144). I set 262144 so that the 16K to 128K context steps fit (forced by my tests)

How well it answers

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-10.

Note from the run: MMLU-Pro (400 questions), GPQA Diamond (198), IFEval (200), LiveCodeBench (40 tasks) and RULER at 8K and 32K tokens (40 tasks each) ran with thinking off and the sampling of the model card (temperature 1.0, top-p 0.95; nothing else is named, so the engine's defaults apply), seed 42. Eight requests ran at once (the server's streams; each question is its own request with the same limits and seed). MMLU-Pro and GPQA Diamond ran with an answer limit of 1,024 tokens (the other pages use 48), letter only; IFEval with 4,096 tokens; LiveCodeBench with 32,768 tokens; RULER with 128 tokens. Without thinking the model answers the multiple-choice questions with a letter at once (median 2 tokens), and 2 MMLU-Pro answers had no letter that could be read (they count as wrong). 1 IFEval answer and 6 LiveCodeBench answers were cut off at their limit and count as wrong; the other tests had none cut off. RULER by task: variable tracking 3 of 10 at 8K and 2 of 10 at 32K; multi-hop question answering 7 of 10 and 5 of 10. The thinking test (GPQA 30) ran with thinking on and a limit of 16,384 tokens: 12 of the 30 answers were cut off at the limit and count as wrong (the model thinks very long: median 9,623 tokens and 77.1 s per question, with 8 questions running at once). Two answer files are not published, because the site's filter for machine names and power readings held them back: the IFEval answers (ifeval.jsonl) and the LiveCodeBench results file (lcb.jsonl). The summaries, the logs and the contracts of both tests are published, and the numbers of the table were counted from the complete files. These quality tests have no foreign-load guard. The Claude Code session and the macOS desktop shell ran on the same Mac. The sampling file (not published) lists the busiest other process every 10 seconds: 38 of 128 samples were above 25 % of one core (up to 182 %): 30 of them the macOS desktop shell (menu bar and control center), 4 Apple's audio service, 3 the virtual machine of the code sandbox of the LiveCodeBench test itself and 1 a macOS metadata worker (the peak). GPQA Diamond: the original is gated and no Hugging Face token was available, so the public mirror fingertap/GPQA-Diamond (198 rows) was used. These are my own tests, and no other run of this setup is mixed into the numbers on this page.

bash tools/quality2/measure_all.sh tools/quality2/cards/nemotron-3.5-lightning-30b-a3b-mlx4bit-tf-1.2.5-official-m5ultra-1010.json --parts mmlupro --job mac256-run-nemo-q

bash tools/quality2/measure_all.sh tools/quality2/cards/nemotron-3.5-lightning-30b-a3b-mlx4bit-tf-1.2.5-official-m5ultra-1010.json --parts gpqa --job mac256-run-nemo-q

bash tools/quality2/measure_all.sh tools/quality2/cards/nemotron-3.5-lightning-30b-a3b-mlx4bit-tf-1.2.5-official-m5ultra-1010.json --parts ifeval --job mac256-run-nemo-q

bash tools/quality2/measure_all.sh tools/quality2/cards/nemotron-3.5-lightning-30b-a3b-mlx4bit-tf-1.2.5-official-m5ultra-1010.json --parts lcb --job mac256-run-nemo-q

bash tools/quality2/measure_all.sh tools/quality2/cards/nemotron-3.5-lightning-30b-a3b-mlx4bit-tf-1.2.5-official-m5ultra-1010.json --parts ruler --job mac256-run-nemo-q

bash tools/quality2/measure_all.sh tools/quality2/cards/nemotron-3.5-lightning-30b-a3b-mlx4bit-tf-1.2.5-official-m5ultra-1010.json --parts ruler --job mac256-run-nemo-q

bash tools/quality2/measure_all.sh tools/quality2/cards/nemotron-3.5-lightning-30b-a3b-mlx4bit-tf-1.2.5-official-m5ultra-1010.json --parts think --job mac256-run-nemo-q

How much context fits

Memory test. The context is fixed at 262,144 tokens by my start command (see the changes in the speed notes), so the server is started once with that command and reserves its memory up front. Then, for each size (4K, 8K, 16K ... up to the largest size tested), one prompt that fills about 95 % of that size is sent, and the model must answer. The largest size that answered counts; it is the highest size tested, not a limit of the setup. Decode speed at that size was not measured. Peak memory is the rise of used memory over the idle value before the server start (unified memory, so GPU and CPU share it). Measured 2026-11-10. Raw files: result of every step · log · contract.

Note from the run: Highest rise over idle: 27.90 GiB. The server was started once, with the context set to 262,144 tokens (the window of my start command, see the changes in the speed notes), and it reserved its memory up front. Then one prompt of about 95 % of each size (4K, 8K, 16K, 32K, 64K and 128K) was sent, and the model had to answer. All six sizes were answered (finish reason stop). The largest size tested is 131,072 tokens (a prompt of 124,518 tokens): it is the highest size I tested, not a limit of the setup. The window of 262,144 tokens was never filled, so there is no result above 128K. The memory number is the rise of the Mac's used memory over its idle value before the server was started (a Mac has one pool of memory for CPU and GPU): the Mac had 23.11 GiB in use at idle; the rise over idle was at most 23.17 GiB while the server loaded, 23.87 GiB during the 4K request and 27.90 GiB during the 128K request. Because it is the rise of the whole Mac, other programs that used or released memory at the same time are inside the number. This engine reports no timings for this memory test, so the decode speed and the first-token columns are empty.

bash tools/quality2/measure_all.sh tools/quality2/cards/nemotron-3.5-lightning-30b-a3b-mlx4bit-tf-1.2.5-official-m5ultra-1010.json --parts fit --job mac256-run-nemo-fit

How it plays the Gauntlet

One solo race, season seed 21, one attempt, no help.

Game rank as of Nov 10, 2026. The rank is the order of the Gauntlet finish times in this memory class. Race page and raw files.

The race ran the recipe named on its race page, with the sampling and answer limit of its lane config. The engine build of the race is not recorded in the race files.

Result (result.json) Race log (log.jsonl) Model answers (io.jsonl) System prompt Exact command Verify output Model config Race page

Also made this possible

• Engine (TensorFold 1.2.5, official release)Ash Hart@ashxhart GitHub Hugging Face Website
• Model (Nemotron 3.5 Lightning 30B-A3B)NVIDIAHugging Face GitHub Website
• MLX 4-bit weights (revision d9d758fb, with the MTP head)TTensorFold (Hugging Face organisation)Hugging Face

Run it yourself

What you need and how to start it, step by step. The results above come from exactly this setup.

What you need

Hardware

Mac Studio M5 Ultra, 256 GB unified memory, one box (Darwin kernel 27.0.0)

System

macOS (the Mac above)

Model

NVIDIA Nemotron 3.5 Lightning 30B-A3B (hybrid Mamba-2 + attention + MoE, 30B total / 3B active, 52 layers), TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit @d9d758fb (MLX affine 4-bit, group 64, community conversion of NVIDIA's BF16 checkpoint) incl. native MTP head

Model (Nemotron 3.5 Lightning 30B-A3B)

huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

MLX 4-bit weights (revision d9d758fb, with the MTP head)

huggingface.co/TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit

Engine

TensorFold 1.2.5, the official macOS release (native Metal engine), MLX 4-bit weights with the MTP drafts of the checkpoint, 262,144 tokens of context (set by me, see the changes below), up to 8 streams (the engine's default) — the program that loads the model and answers requests · github.com/ashhart/TensorFold/releases/tag/v1.2.5

Recipe (docs/recipes/nemotron-3.5.md)

github.com/ashhart/TensorFold/blob/main/docs/recipes/nemotron-3.5.md by Ash Hart

Steps

1. Get the recipe, the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   tensorfold-native serve \
     --host 127.0.0.1 \
     --port 8888 \
     --context 262144 \
     --keep-warm 86400 \
     --no-update-check nemotron-lightning

   The file names are the ones we used. Run the command in the folder that holds them, or put the full path in front of each name.

   What the parts mean

   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on

TensorFold 1.2.5 official release archive macos-arm64 (tensorfold-native, tag v1.2.5), native Metal Nemotron family with MTP drafts, --context 262144, --parallel auto, prompt cache 16 GiB default, text only; tensorfold-native serve --host 127.0.0.1 --port 8888 --context 262144 --keep-warm 86400 --no-update-check $HOME/models/nemotron-lightning (Ash's recipe docs/recipes/nemotron-3.5.md, section Run, plus the changes named on this page; the folder holds the files of the Hugging Face repository at revision d9d758fb)