Ash's recipe: Qwen3.8-Flash-Next 6-bit on a Mac Studio M5 Ultra (TensorFold 1.2.8)

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-15. Raw files: speed result · speed contract · log · long answer.

What ran. Ash's TensorFold 1.2.8, the official release for macOS (arm64, no unmerged pull request, no TF_FLASHNEXT switch), serving Qwen3.8-Flash-Next as the MLX 6-bit checkpoint with the MTP head (TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP, revision 45032a19), with the flags of the recipe's line for the M5 Ultra: --parallel 2, --vision and an 8 GiB prompt cache. Where this run differs from the recipe. (1) Window: 262,144 tokens instead of the recipe's 1,048,576. The engine plans 238.59 GiB for the 1,048,576 window, and the load limit of the recipe's line (FLASHNEXT_LOAD_LIMIT_GB=240) is 223.52 GiB, so it does not load on this 256 GB machine; with 262,144 the plan is 193.15 GiB and the model loads in 46.6 s the first time and 28.6 s afterwards. (2) Presence penalty: the speed test's instruct profile uses presence penalty 1.5 elsewhere; here it is 0 because Metal refuses any other value. (3) Concurrency: the server has 2 sessions, so the levels 4 and 8 at once are above its slot count and queue. The recipe lists one stream at 142 to 214 tok/s for this window; the speed test here (other prompts, sampling temperature 0.7, thinking off) measures a decode between 114.7 and 142.7 tok/s depending on the context depth. Some speed steps were run again before the run that this page shows: four earlier attempts were stopped by foreign processes or by a request that ended early; their raw files are not part of this page. The 64K context step shows a code decode value of 225.8 tok/s that is far above the others: in that run the request starts with a tool call that repeats the input code word for word, so the copy drafts accept almost every token. Do not read it as a normal value.

python3 speed2.py --base-url http://127.0.0.1:53065/v1 --label flashnext6-mac-tf128-1511-s-speed-r4 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $HOME/bench/speed2 --contract $HOME/bench/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/models/Qwen3.8-Flash-Next-MLX-6bit-MTP/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0} --sampling-source https://huggingface.co/Qwen/Qwen3.8-Flash-Next README.md lines 340 (Thinking) and 341 (Instruct/non-thinking) at revision 7ecef55a, unchanged in these lines at main of Nov 7, 2026, local copy not published (same as the other Flash Next cards); the checkpoint generation_config.json (1.0/20/0.95) = thinking default. Ash's recipe command sets --temperature 0 (docs/recipes/qwen3.8-flash-next.md:19), overridden per request by the card sampling, which the API allows. Seed 42 for speed/quality. || FORCED DEVIATION Nov 15, 2026: the model card's instruct presence_penalty 1.5 is set to 0.0, because the TensorFold 1.2.8 Metal engine (backend 'flashnext-zig') refuses nonzero values: server answer 'presence_penalty and frequency_penalty are not supported by backend 'flashnext-zig': send 0 or leave them out', zig/src/server/server.zig:335 at v1.2.8, docs/api.md:373 at v1.2.8 ('other engines refuse nonzero values with 400; zero ... keep working'), open issue #3113. temperature 0.7, top_p 0.8, top_k 20, min_p 0 stay as on the model card. Thinking set unchanged (its presence_penalty is 0). --model bench --no-cache-prompt --server-log $HOME/bench/server.log --engine-version-file $HOME/bench/engine-version.txt --server-slots 2 --server-slots-source server start log (1.2.8, --context 262144 --parallel 2 --vision --prompt-cache-gib 8), file $HOME/bench/server.log line 8: 'info: Flash Next: 2 session slots (262144,262144 tokens), session budget 41.2 GiB; prompt cache 8.0 GiB, shared target forwards' (read Nov 15, 2026 00:29); explicit count from Ash's M5-Ultra line docs/recipes/qwen3.8-flash-next.md:137; --parallel auto (default) would use the single-reply driver (docs/recipes/qwen3.8-flash-next.md:98) --server-ctx 262144 --server-ctx-source recipe docs/recipes/qwen3.8-flash-next.md:137-138 and :141-142 at v1.2.8

How well it answers

Thinking off in the first five tests (enable_thinking false in every request), seed 42, 2 questions at once. Answer limit 1,024 tokens for MMLU-Pro and GPQA. GPQA 30 with thinking on uses the model card's thinking sampling and a limit of 16,384 tokens. Raw numbers only, no overall score.

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-15.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-flashnext-mlx6bit-mtp-tf-1.2.8-official-par2-m5ultra-1511.json --parts mmlupro --job flashnext6-mac-tf128-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-flashnext-mlx6bit-mtp-tf-1.2.8-official-par2-m5ultra-1511.json --parts gpqa --job flashnext6-mac-tf128-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-flashnext-mlx6bit-mtp-tf-1.2.8-official-par2-m5ultra-1511.json --parts ifeval --job flashnext6-mac-tf128-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-flashnext-mlx6bit-mtp-tf-1.2.8-official-par2-m5ultra-1511.json --parts lcb --job flashnext6-mac-tf128-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-flashnext-mlx6bit-mtp-tf-1.2.8-official-par2-m5ultra-1511.json --parts ruler --job flashnext6-mac-tf128-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-flashnext-mlx6bit-mtp-tf-1.2.8-official-par2-m5ultra-1511.json --parts ruler --job flashnext6-mac-tf128-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-flashnext-mlx6bit-mtp-tf-1.2.8-official-par2-m5ultra-1511.json --parts think --job flashnext6-mac-tf128-1511-t

How much context fits

Memory test. The recipe fixes the context (262,144 tokens), so the server is started once with the recipe's own command and reserves its memory up front. Then, for each size (4K, 8K, 16K ... up to the largest size tested), one prompt that fills about 95 % of that size is sent, and the model must answer. The largest size that answered counts; it is the highest size tested, not a limit of the setup. Decode speed at that size was not measured. Peak memory is the rise of used memory over the idle value before each server start (unified memory, so GPU and CPU share it). Measured 2026-11-15. Raw files: result of every step · log · contract.

Note from the run: fit memory per node, highest rise over idle: local 188.6 GiB

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-flashnext-mlx6bit-mtp-tf-1.2.8-official-par2-m5ultra-1511.json --parts fit --job flashnext6-mac-tf128-1511-t

Also made this possible

• Engine (TensorFold 1.2.8, official release)Ash Hart@ashxhart GitHub Hugging Face Website
• ModelQwen@Alibaba_Qwen Hugging Face GitHub Website
• Model files (MLX 6-bit with MTP head, revision 45032a19); Hugging Face organisation TensorFold, whose only listed member is ashxhartTensorFoldHugging Face @ashxhart GitHub Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

Mac Studio M5 Ultra, 256 GB unified memory

System

macOS (the Mac above)

Model

Qwen/Qwen3.8-Flash-Next (qwen4_exp: Gated DeltaNet, sparse attention, MoE, hyper-connections, n-gram embeddings), TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP @45032a19 (MLX affine 6-bit, group 32, native MTP head)

Model files (MLX 6-bit with MTP head, revision 45032a19); Hugging Face organisation TensorFold, whose only listed member is ashxhart

huggingface.co/TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP

Engine

TensorFold 1.2.8 official release (macOS arm64) — the program that loads the model and answers requests · github.com/ashhart/TensorFold/tree/v1.2.8

Recipe (docs/recipes/qwen3.8-flash-next.md at v1.2.8)

github.com/ashhart/TensorFold/blob/v1.2.8/docs/recipes/qwen3.8-flash-next.md by Ash Hart

Steps

1. Get the recipe, the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   env FLASHNEXT_LOAD_LIMIT_GB=240 tensorfold-native serve TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP \
     --name bench \
     --host 127.0.0.1 \
     --port 8888 \
     --context 262144 \
     --parallel 2 \
     --vision \
     --prompt-cache-gib 8 \
     --temperature 0 \
     --keep-warm 86400 \
     --no-update-check

   What the parts mean

   • --parallel 2how many requests it answers at the same time
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on
   • --name benchthe model ID the server advertises to clients

TensorFold 1.2.8 official release archive macos-arm64 (tensorfold-native, tag v1.2.8), native MTP drafts, --name bench, --context 262144, --parallel 2 --vision --prompt-cache-gib 8; env FLASHNEXT_LOAD_LIMIT_GB=240 tensorfold-native serve TensorFold/Qwen3.8-Flash-Next-MLX-6bit-MTP --name bench --host 127.0.0.1 --port 8888 --context 262144 --parallel 2 --vision --prompt-cache-gib 8 --temperature 0 --keep-warm 86400 --no-update-check (recipe line docs/recipes/qwen3.8-flash-next.md lines 17 to 20 and the M5 Ultra line at lines 137 and 138, v1.2.8; the official release binary tensorfold-1.2.8-macos-arm64; context 262144 instead of the recipe's 1048576)