Incept5's inferd 0.4.1: Qwen3.8-Flash-Next (oQ4e, MTP) on a Mac Studio

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-10. Raw files: speed result · speed contract · log.

The speed test ran with the start command of the author of the engine, Incept5 (GitHub organisation Incept5-AI): the agents-at-once command of examples/agents-at-once/README.md, line 22 at tag v0.4.1, MODEL=<model folder> MTP=1 ./run.sh 8300 128 2048 64 256 65536. Its arguments are the port (8300), the agents running at once (128), the step (2,048 tokens), the KV pool (64 GiB), the snapshots (256) and the window (65,536 tokens, prompt plus output). The README of the engine names the Mac Studio with 256 GB for the option --model flash-next (line 102) and the author's machine list names the M5 Ultra with 256 GB as its main development and benchmark machine (docs/MACHINES.md, line 11). The weights are the oQ4e requant of Qwen3.8-Flash-Next with the MTP head by Jundot (Hugging Face Jundot/Qwen3.8-Flash-Next-oQ4e-mtp, revision 2322c888, 29 Oct 2026, 'Re-quantize with the improved oQe quantization in oMLX 0.7.3'; 34 files, 100.30 GiB), not a quant of mine; the author's own model script names this repository for Flash-Next and says that every Flash-Next result of the author's results used this oQ4e build (engine/tools/models.zsh, lines 19 and 34). I downloaded the files file by file at that revision and checked the sha256 of the 22 large files against the Hugging Face listing (all 22 matched, 9 Nov 2026, 22:32); no file of the folder was changed afterwards. The engine is the official release archive inferd-0.4.1-macos27-arm64.tar.gz (tag v0.4.1, published on 5 Nov 2026 at 16:12 UTC; sha256 39b1aa08; commit 4ad756ad; its binary inferd, sha256 cd16f8d8), not built by me. Release v0.4.1 is the newest release of the repository (read on 11 Nov 2026). The repository's main branch is further ahead: its newest change (8 Nov 2026, 15:53 UTC) changes scheduler defaults; no test on this page used it.

Differences from the author's recipe, all named: (1) A local relay. inferd answers every request that carries a cache_salt field with HTTP 400 (engine/frontend/serve.py, lines 1057 to 1059), and the speed test tool sends that field in its requests. A small relay on port 8888 sits in front of the server (which stays on the author's port 8300); it deletes exactly that one field and changes nothing else. Every prompt starts with a unique nonce, so no request shares a prompt prefix with another, and the tool's cache check found no cached prompt tokens in any of the 52 requests it examined. The relay is not part of the author's recipe, and I did not test whether it changes the numbers. (2) Sampling. The author's own harness sends temperature 0 and thinking off (bench/studio/conc.py, lines 27 to 33; thinking is off by default, line 24), so the speed test and the instruction tests use temperature 0 and seed 42 in every request (the one thinking test is described with the quality tests). The instruct values of the Qwen model card (temperature 0.7, top-p 0.80, presence penalty 1.5; model card, line 342) are not used: inferd sends a request with a presence penalty to the host sampler and never speculates it (engine/frontend/serve.py, lines 2028 to 2032), so the MTP head would not draft, which is not how the author runs the engine. Top-k and top-p come from the checkpoint's generation config (top-k 20 and top-p 0.95, engine/frontend/serve.py, line 130); with temperature 0 they have no effect. Everything else of the command is the author's: the model folder, MTP=1 and the six arguments. The load guard's ignore list (named in the speed contract) holds the process that controls the tests, file-search tools, the macOS desktop shell, applications under /Applications, the dictation helper of this Mac and the three processes of this setup (the engine, its front end and the relay), so they are not counted as foreign load. The author's published figures come from other tests, for example the agents-at-once test of the README headline (README, line 132: 64 agents at once on 16K contexts); this page makes no comparison with them. I ran that author script three times as a demonstration on this Mac (10 Nov); it is not one of the measurements of this page, so no number of it is shown. Text only: no vision, no tool use and no agent task is measured.

The headline 127.8 tok/s is the mean of the code prompt (135.0) and the prose prompt (120.5), medians of two runs each; the spread (CV) of the two code runs is 5.3 % and of the two prose runs 2.3 % (both under the 15 % limit of our tool). Thinking was off in every request. Requests at once: 1, 2, 4 and 8 requests ran together; the server has 128 slots, so no request waited for a slot. The total rises from 108.3 tok/s with 1 request to 153.7 with 2, 207.3 with 4 and 291.5 with 8, and the speed of one request falls from 108.3 to 37.2 tok/s with 8 requests; the per-request figure is measured over the whole request, so it is below the decode speed. Long prompts: the table shows 16K and 32K. The 64K and 128K steps are not shown, and the reason is the window: the tool refuses a step when the prompt and its 512 output tokens do not fit into the server's window, and 65,536 + 512 = 66,048 is more than the 65,536 tokens of the author's command (the 128K step does not fit either). The prefill at 64K and 128K is therefore not measured with the author's command, and this page shows no number for it. To get 64K and 128K numbers I ran a supplement with the window changed to 131,072 tokens (the sixth argument of run.sh), which is not the author's recipe. Its memory test is registered and described in the memory test below. Its speed test did not produce valid 64K and 128K values. In 5 of 6 attempts (10 Nov 22:57 to 11 Nov 07:33) one code run of the 32K or 64K step ended after only 81, 81, 95, 95 or 111 tokens (the tool needs at least 256 tokens to count a run); in one of these attempts the 128K step was also lost to foreign load (a macOS service, duetexpertd), and the sixth attempt stopped at the 64K step because of foreign load (an ffmpeg job). An earlier supplement run (10 Nov, 01:37 to 01:45) was valid at all four depths, but it was not entered in the register, so no number of it is shown here, and the logs of all these attempts are not published.

The server was started four times between 23:50 on 9 Nov and 00:18 on 10 Nov before the speed run (start lines in speed2.json: 23:50, 00:06, 00:08 and 00:17). My notes of that night say that the first start after the download behaved abnormally (a simple request took far longer than on later starts); I could not determine the cause and restarted the server. From the third start on, the start script sends one probe request after every start (a health check, not part of the measurements; 68.2 tok/s at 00:08 and 68.6 tok/s at 00:18) and restarts the server if it answers below 40 tok/s. A probe run from 00:08 to 00:17 was stopped, produced no values and is not part of this page. The speed test ran from 00:18 to 00:22 on the server started at 00:17; the instruction tests and the thinking test ran on another fresh start (00:22), LiveCodeBench on a third (00:44), all with the same command. I ran the author's release gate (tools/release_gate.sh, README line 119, shown there with the 27B model) with these Flash-Next weights before the speed test: its host, front-end and sampler tests passed, and its logits check against mlx_lm did not: it compares four sequences with a threshold of cosine 0.995 and KL 0.03 (0.99 and 0.05 for sequence 0), 1 of 4 passed at chunk size 1 and the gate stopped there; with the same commands by hand 2 of 4 passed at chunk size 24, 1 of 4 at 256 and 1 of 4 at 1,536. The top token was right in all 16 comparisons, the cosine values lie between 0.990 and 0.997 (lowest per chunk size: 0.991, 0.990, 0.994 and 0.994), and they are not lower at the chunk sizes 256 and 1,536 than at 1 and 24. The gate fails formally; this page draws no further conclusion from it.

Load on the Mac: the load guard (a sample every half second, 483 samples) repeated 1 of its 29 steps and threw away 1 try: the third prefill run at 32K, where Apple's audio service (coreaudiod) used up to 26.1 % of one core for two samples in a row; it ran again after 5 s of quiet, and the repeated try peaked at 23.6 %. The guard stops a step when one foreign process stays above 25 % of one core for two samples in a row. In 14 other steps a process was above 25 % in single samples only, so those steps stand; the busiest process of each of them: decode, prose prompt, run 1: a Python process, 110.0 %; decode, code prompt, run 2: Apple's audio service (coreaudiod), 52.7 %; prefill at 8K, run 1: the Metal shader compiler service (MTLCompilerService), 44.0 %; prefill at 8K, run 2: Spotlight's metadata store (mds_stores), 30.0 %; prefill at 32K, run 1: Apple's audio service (coreaudiod), 34.8 %; 1 request at once, round 1: Apple's audio service (coreaudiod), 29.7 %; 2 requests at once, round 2: Apple's audio service (coreaudiod), 48.7 %; 4 requests at once, round 1: a Python process, 56.2 %; 8 requests at once, round 1: Apple's audio service (coreaudiod), 48.5 %; 8 requests at once, round 2: Apple's audio service (coreaudiod), 54.0 %; 16K long prompt, prose, run 1: Apple's audio service (coreaudiod), 53.3 %; 32K long prompt, code, run 1: Apple's audio service (coreaudiod), 51.9 %; 32K long prompt, prose, run 1: Safari's SafeBrowsing service, 55.2 %; 32K long prompt, prose, run 2: Apple's audio service (coreaudiod), 69.6 %. In the remaining 15 steps (the repeated one among them, in its repeated try) no process was above 25 %. The quiet check (one sample every 10 seconds, 24 samples) saw 1 sample above 25 % of one core: launchd, 28.4 %. The quiet logs before and after the run list no foreign problems. The test tool holds three indexer processes (media analysis, photo analysis and Spotlight indexing) while it runs and resumes them at its end; no foreign process was ended. So the Mac was not idle, and I cannot show that nothing else used it during the tests: the numbers are measured under this load. Not published: the quiet logs and the sampling file of this run (the site's filter holds back files of that kind by name), the load guard's sample file and error file, and the per-request capture folder (requests/). So the foreign-load figures above can be checked only in part: the speed contract, with the summaries of the load guard and of the quiet check and its record of each step, is published, while the guard's sample file, the quiet logs and the sampling file are not. In the published raw files the source notes (the line numbers of the author's files, the names of our machines and folders, a few German words) were corrected or made neutral; no measured value was touched.

python3 speed2.py --base-url http://127.0.0.1:64364/v1 --label inferd041-mac-0911-s-speed-r0 --prompts $BENCH/repos/speed-test-v2/prompts-v2.json --output $RUN/inferd041-mac-0911-s-speed-r0/inferd041-mac-0911-s-speed-r0/speed2 --contract $RUN/inferd041-mac-0911-s-speed-r0/inferd041-mac-0911-s-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/models/Qwen3.8-Flash-Next-oQ4e-mtp/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.0} --sampling-source instruct = the author's own harness: temperature 0 (greedy), thinking off, no other sampling field (bench/studio/conc.py:27-33 at tag v0.4.1, function body(): enable_thinking False, temperature 0; the agents-at-once and quick64 runs use it). Reason not to take the Qwen card instruct values (temperature 0.7, top_p 0.8, presence_penalty 1.5): inferd sends any request with a non-neutral presence_penalty to the host sampler and never speculates it (frontend/serve.py:2028-2032), i.e. the MTP head would be off, which is not how the author runs the engine. top_k / top_p are omitted and come from the checkpoint's generation_config (top_k 20, top_p 0.95, serve.py:129-132); with temperature 0 they have no effect. thinking = the Qwen3.8-Flash-Next model card (the recommended sampling parameters in the model card of Qwen/Qwen3.8-Flash-Next, thinking mode: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0) because the author's harness has thinking off by default (bench/studio/conc.py, line 24 at tag v0.4.1); penalty 0 and top_k 20 <= 64 keep the speculative path. cache_salt: inferd answers HTTP 400 to any cache_salt (frontend/serve.py:1057-1059: its prefix cache is shared, no salts) and speed2 sends one on every request; a small local relay (port 8888 -> inferd 8300) deletes exactly that one key and nothing else (it counts the requests it changed in a file that is not published); the unique nonce at the start of every first message keeps each prompt cold, cached_tokens in the raw SSE are checked. Seed 42 for speed and quality. --model qwen3.8-27b --no-cache-prompt --server-log $LOGS/engine/inferd-8300.log --engine-version-file $LOGS/engine-version.txt --server-slots 128 --server-slots-source run.sh positional argument 2 (max_running) = 128 in the author's command (examples/agents-at-once/README.md line 22 at tag v0.4.1; engine/run.sh:164 positional form); 256 snapshots, KV 64 GiB, step 2048 --server-ctx 65536 --server-ctx-source run.sh positional argument 6 (max context) = 65536 in the author's command (examples/agents-at-once/README.md:22); the Flash-Next default of run.sh is also 65536 (engine/run.sh:134 at tag v0.4.1). The limit covers prompt plus output tokens.

How well it answers

Thinking off, except the last test. Seed 42. Sampling as the author runs it (temperature 0), see the note below. A cut-off answer counts as wrong. Measured 2026-11-10.

Note from the run: MMLU-Pro (400 questions), GPQA Diamond (198), IFEval (200), LiveCodeBench (40 tasks) and RULER at 8K and 32K tokens (40 tasks each) ran with thinking off and temperature 0 (the author's harness, see the speed notes), seed 42. 128 requests ran at once (the server's 128 agents; each question is its own request with the same limits and seed); the thinking test had its 30 questions at once. MMLU-Pro and GPQA Diamond ran with an answer limit of 1,024 tokens (the other pages use 48), letter only; IFEval with 4,096 tokens; LiveCodeBench with 32,768 tokens; RULER with 128 tokens. Most answers to the multiple-choice questions are a single letter (median 2 tokens); every answer except the one cut-off GPQA answer (no letter extracted) had a letter that the tool could read. 0 MMLU-Pro answers and 1 GPQA answer were cut off at the 1,024-token limit (counted wrong); 1 IFEval answer and 1 LiveCodeBench answer were cut off at their limits (counted wrong), and the long-text tests had none cut off. RULER by task at 8K and 32K: multi-key retrieval 10 of 10 and 10 of 10; common-word extraction 10 of 10 and 10 of 10; multi-hop question answering 8 of 10 and 3 of 10; variable tracking 0 of 10 and 0 of 10. Variable tracking asks for all variables that hold a value; the gold answer has four names, the model named one name in 17 of the 20 replies and two in 3, never all four, and the tool counts only a complete answer. Three of the 80 long-text prompts were one token off the target length (8,191, 8,193 and 32,767 tokens; the tool counts with the model's own tokenizer). IFEval is scored with the official checker of Google Research (instruction_following_eval, commit 3ef52111), strict, at the level of the prompt. LiveCodeBench: the 40 newest tasks of the tool's list (code_generation_lite, release test6; tasks dated 2025-03-15 to 2025-04-06), pass at 1, the code runs in a container without network access; 3 answers held no code block that the tool could extract (counted wrong), and no task was lost to a sandbox error. The thinking test (GPQA 30) ran with thinking on and a limit of 16,384 tokens (thinking and answer together), with the thinking values of the Qwen model card (temperature 1.0, top-p 0.95, top-k 20, presence penalty 0; model card, line 341), because the author's harness has thinking off: 8 of the 30 answers were cut off at the limit and count as wrong; the model thinks long: median 5,607 tokens and 253.6 s per question. These quality tests have no foreign-load guard. The sampling files (not published, see the speed notes) list the busiest other process every 10 seconds: 21 of 191 samples were above 25 % of one core: 3 a macOS job that compresses the Wi-Fi log file (bzip2) (up to 99.0 %), 7 Apple's audio service (coreaudiod) (up to 61.1 %), 2 the virtual machine of the code sandbox of the LiveCodeBench test itself (up to 104.7 %), 2 the network extension of a VPN application (up to 41.0 %), 1 launchd (34.6 %), 1 a macOS display-information tool (system_profiler) (32.2 %), 1 a macOS metadata worker (27.8 %), 1 Apple's iCloud sync service (cloudd) (32.5 %), 1 a Python process (30.8 %), 1 the Metal shader compiler service (25.8 %), 1 Apple's network name service (mDNSResponder) (32.0 %). The quality contracts record the peak of each part: 99.0 % in the instruction tests, 61.1 % in the thinking test and 104.7 % in LiveCodeBench. The quiet check after the instruction tests and the one after the thinking test each list one foreign process, the macOS kernel (kernel_task) at 44 % of one core at that moment; the other quiet logs list none. GPQA Diamond: the original is gated and no Hugging Face token was available, so the public mirror fingertap/GPQA-Diamond (198 rows) was used. Not published, because the site's filter held them back: the results file and generations of LiveCodeBench (lcb.jsonl, lcb.gen.jsonl), the folders with the extracted programs and the sandbox runs (lcb/code, 76 files; lcb/sandbox, 144 files), the 80 prompt files of RULER (ruler/prompts), and the quiet logs and sampling files of all parts. The summaries and logs of all tests, the quality contracts and the answers of MMLU-Pro, GPQA, IFEval, RULER and the thinking test are published, and the numbers of the table were counted from the complete files. These are my own tests, and no other run of this setup is mixed into the numbers on this page.

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-oq4e-mtp-inferd-0.4.1-m5ultra-0911.json --parts mmlupro --job inferd041-mac-0911-q

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-oq4e-mtp-inferd-0.4.1-m5ultra-0911.json --parts gpqa --job inferd041-mac-0911-q

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-oq4e-mtp-inferd-0.4.1-m5ultra-0911.json --parts ifeval --job inferd041-mac-0911-q

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-oq4e-mtp-inferd-0.4.1-m5ultra-0911.json --parts lcb --job inferd041-mac-0911-ql

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-oq4e-mtp-inferd-0.4.1-m5ultra-0911.json --parts ruler --job inferd041-mac-0911-q

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-oq4e-mtp-inferd-0.4.1-m5ultra-0911.json --parts ruler --job inferd041-mac-0911-q

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-oq4e-mtp-inferd-0.4.1-m5ultra-0911.json --parts think --job inferd041-mac-0911-q

How much context fits

Memory test. The window is fixed at 65,536 tokens by the author's start command (the sixth argument of run.sh), so the server is started once with that command. Then, for each size (4K, 8K, 16K ... up to the largest size tested), one prompt that fills about 95 % of that size is sent, and the model must answer. The largest size that answered counts; it is the highest size tested, not a limit of the setup. Decode speed at that size was not measured. Peak memory is the rise of used memory over the idle value before each server start (unified memory, so GPU and CPU share it). Measured 2026-11-10. Raw files: result of every step · log · contract.

Note from the run: The highest rise over idle at any step was 226.76 GiB, at the 32K step; the peak above is the one of the largest size tested. The server was started once, with the author's command, and all five sizes (4K, 8K, 16K, 32K and 64K) were answered (finish reason stop). The largest size tested is 65,536 tokens (a prompt of 62,259 tokens, about 95 % of the size), the whole window of that command, so this test could not go further without a different window. The memory number is the rise of the Mac's used memory over its idle value before the server was started (a Mac has one pool of memory for CPU and GPU): the Mac had 17.17 GiB in use at idle; the rise over idle was at most 225.76 GiB while the server loaded, and the peaks of the five requests lie between 225.69 GiB (16K) and 226.76 GiB (32K); the 4K request peaked at 225.74 GiB and the 64K request at 225.86 GiB. Because it is the rise of the whole Mac, other programs that used or released memory at the same time are inside the number. This engine reports no timings for this memory test, so the decode speed and the first-token columns are empty. Memory test with a larger window (not the author's command): to learn what the setup does with 128K-token prompts, I started the server again with the window changed to 131,072 tokens (the sixth argument of run.sh; nothing else changed) and ran the memory test the same way (10 Nov 2026, 23:11 to 23:12; raw files in the folder fit131k-r0 of this page). All six sizes (4K to 128K) were answered; the largest prompt had 124,518 tokens, and the rise over idle at that size was 225.09 GiB; the highest rise of any step was 225.26 GiB, at the 16K step. This window is not the author's, so it is not the setup of the tables on this page. The test tool's server logs of the memory tests are not published (the site's filter holds them back); fit.json, the output of each test and its contract are published.

bash tools/quality2/measure_all.sh tools/quality2/cards/flashnext-oq4e-mtp-inferd-0.4.1-m5ultra-0911.json --parts fit --job mac256-run-fn-fit

How it plays the Gauntlet

One solo race, season seed 21, one attempt, no help.

Game rank as of Nov 10, 2026. The rank is the order of the Gauntlet finish times in this memory class. Race page and raw files.

The race ran the recipe named on its race page, with the sampling and answer limit of its lane config. The engine build of the race is not recorded in the race files.

Result (result.json) Race log (log.jsonl) Model answers (io.jsonl) System prompt Exact command Verify output Model config Race page

Also made this possible

• Engine (inferd 0.4.1, official release)Incept5GitHub Website
• Model (Qwen3.8-Flash-Next)QwenHugging Face @Alibaba_Qwen GitHub Website
• oQ4e weights with MTP head (oQe requant, revision 2322c888)Jundot (Jun Kim)@jundotkim Hugging Face GitHub Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

Mac Studio M5 Ultra, 256 GB unified memory, one box

System

macOS (the Mac above)

Model

Qwen3.8-Flash-Next oQ4e-mtp (Jundot, oMLX 0.7.3 oQe requant @2322c888)

Model (Qwen3.8-Flash-Next)

huggingface.co/Qwen/Qwen3.8-Flash-Next

oQ4e weights with MTP head (oQe requant, revision 2322c888)

huggingface.co/Jundot/Qwen3.8-Flash-Next-oQ4e-mtp/tree/2322c888c2c314d4eceddf0159dec5d821aa1e41

Engine

inferd 0.4.1, the official macOS release (native Metal engine with an OpenAI front end), Jundot's oQ4e-mtp weights, the start command of the author's agents-at-once example: up to 128 agents, a window of 65,536 tokens — the program that loads the model and answers requests · github.com/Incept5-AI/inferd/releases/tag/v0.4.1

Recipe (start command of examples/agents-at-once, line 22)

github.com/Incept5-AI/inferd/blob/v0.4.1/examples/agents-at-once/README.md by Incept5

Steps

1. Get the recipe, the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory.

   Copy

   MODEL=$HOME/models/Qwen3.8-Flash-Next-oQ4e-mtp \
     MTP=1 \
     PY=$HOME/.venvs/inferd/bin/python ./run.sh 8300 128 2048 64 256 65536

inferd 0.4.1 (Incept5-AI/inferd, official release archive macos27-arm64, native Metal engine + OpenAI front end), MTP drafts + copy drafts, 128 agents, KV 64 GiB, 256 snapshots, max context 65536; MODEL=$HOME/models/Qwen3.8-Flash-Next-oQ4e-mtp MTP=1 PY=$HOME/.venvs/inferd/bin/python ./run.sh 8300 128 2048 64 256 65536 (the author's command, examples/agents-at-once/README.md, line 22 at tag v0.4.1: port, agents, step, KV GiB, snapshots, window; the model folder holds the files of Jundot/Qwen3.8-Flash-Next-oQ4e-mtp at revision 2322c888; I put a small local relay on port 8888 in front of the server, see the changes below)