Qwen3.8-27B on two RTX 5060 Ti: Ash's frozen Python engine v0.6.6 with our own patch, window 109,511 tokens, tensor parallel 2

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer.

What ran. Own patch on the frozen Python engine v0.6.6, not an official TensorFold release: Ash's TensorFold Python engine at its tag v0.6.6 (commit cb2ebf0), the version whose recipe we followed; the engine is frozen at that version. The own patch is ours, not Ash's. It changes: each rank loads only its slice of the weights, and the start estimate of the window for two ranks counts the attention cache the way the engine already counts it on one card, so the window grows from 16,384 to 109,511 tokens; rank 1 also frees the buffers of the last request before the next one is allocated. It is not published and it is not part of any TensorFold release. The model is Qwen3.8-27B (a dense model) in the MLX 4-bit checkpoint TensorFold/Qwen3.8-27B-MLX-4bit (revision 22d8d538), with the DFlash2 drafter z-lab/Qwen3.8-27B-DFlash2 (revision 50307d4c), served on two RTX 5060 Ti with tensor parallel 2: rank 1 is started first, then rank 0, as the recipe says (docs/recipes/qwen3.8-27b.md at v0.6.6, lines 135 to 142). The software is torch 2.14.1 for CUDA 13 and Python 3.13.5, with driver 595.71.05.

Where this run differs from the recipe. (1) Engine: the own patch on the frozen v0.6.6 (see above), not the author's code. (2) Window: 109,511 tokens (prompt and reply together), where the recipe's native window is 262,144. The recipe command sets no --context; the engine's own start estimate gives the window, here with our own patch. With TENSORFOLD_MEMORY_RESERVE_GIB=2, the memory reserve that the RUNBOOK of v0.6.6 names (line 117) and that docs/recipes/cuda.md describes (lines 206 to 208), and that the recipe command does not set, the start line reads 'CUDA rank 0 startup estimate 13.13 GiB within 13.13 GiB; native 262144, allocated prompt/reply window 109511, cache slots 109523'. The first page of this setup, with the unpatched estimate, ran with a window of 16,384 tokens. (3) Streams: one. The engine serves one request at a time on CUDA (TensorFold v0.6.6, src/tensorfold/cli.py line 267 and docs/recipes/cuda.md line 197), and the recipe command sets no --parallel. (4) Sampling: the speed test sends the instruct profile of the model card (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence_penalty 1.5) with every request and turns thinking off; the server's own default shown in its start line (temperature 1.0, top_k 20, top_p 0.95) is overridden per request, which the API allows. The recipe command sets no sampling flag. (5) The drafts are on (the start line reads 'drafts: on'). (6) Environment: TENSORFOLD_MEMORY_RESERVE_GIB=2 (see (2)) and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which docs/recipes/cuda.md (lines 209 and 210) names as a way to reduce fragmentation near the memory cap; the recipe command sets neither. (7) Addresses: both ranks run on one machine, so master and host are 127.0.0.1 (the recipe shows 192.0.2.1 and 0.0.0.0).

Window. The window is 109,511 tokens and the server runs 1 stream; the start line records 'context: 109511' and 'loaded in 231.5s'. The engine serves one request at a time, so the levels 2, 4 and 8 at once are above its slot count.

Memory. Right after loading, each card holds 8,950 MiB. In the run itself, card 0 showed up to 14,234 MiB (the largest of the readings taken about every 10 seconds). In separate probes of the same build, with expandable_segments:True a series of prompts of up to 108,810 tokens peaked at 13,648 MiB (13.3 GiB, nvidia-smi) per card, and without that setting four prompts of 104,100 to 104,977 tokens peaked at 15,336 MiB (14.98 GiB); no request in the probes behind these two peak values ran out of memory (two earlier probe starts did run out of memory in a long request, on one or both cards). The note in the published raw files says 'peak 13.6 GiB per card'; that is a unit slip of our note (13,648 MiB divided by 1,000 instead of 1,024 gives 13.6; the correct value is 13.3 GiB). The raw files are unchanged. None of these memory readings is part of the published raw files.

Requests at once. The server has 1 stream: the requests wait in a queue and run one after another, not in parallel. The total stays near the value of a single request (66.7, 67.8, 68.2 and 68.2 tok/s with 1, 2, 4 and 8 requests), and the speed of each request falls as the queue grows (66.7, 50.7, 28.4 and 15.3 tok/s). All requests carry the same prompt.

Dropped step (reason: the server window of 109,511 tokens): the context step at 128K was not run because it cannot fit; the prefill series stops at 64K. The tool checked the 128K size before it sent anything: a prompt of 131,072 tokens plus the 512 reply tokens (131,584) is larger than the 109,511-token window that the server reports, so no request was sent for it and it has no value (the register row speed2_ctx_d131072_fits is 0). The depths 16K, 32K and 64K were measured with two runs of the code prompt and two of the prose prompt each; the prefill series (3 runs each) has 2K, 8K, 16K, 32K and 64K. The prefill series and the context steps are separate requests, so their values at the same depth differ slightly (at 32K: 855.2 tok/s in the prefill series, 850.3 tok/s in the context step). The prefill series at 64K measured 795.3 tok/s with a first token after 82.4 s (register rows speed2_prefill_d65536_tok_s and speed2_ttft_d65536_s); the 16K step of the prefill series (887.6 tok/s, first token after 18.5 s in the raw file) has no register row, and the long-prompt table shows the 16K value of the context step.

Long answer test. The test asks for one reply of up to 32,768 tokens and fits the window of 109,511 tokens. The reply ended on its own (finish reason stop) after 7,326 tokens and took 53.4 s; the output file l1-output.txt holds the reply.

The headline 96.2 tok/s is the mean of the code prompt (108.8) and the prose prompt (83.7), medians of two runs each (spread (CV) of 3.0 % and 5.6 %). The code prompt has 1,258 and 1,259 tokens and the prose prompt 1,355 and 1,356 tokens, and every reply is 512 tokens long and ends at the limit. Acceptance rate and mean accepted length: not available (in one-stream mode of this build the engine does not report them, so the test counted no drafted tokens).

Sampling. The speed test sent temperature 0.7, top_p 0.8, top_k 20, min_p 0 and presence_penalty 1.5 with every request (the instruct profile of the model card), seed 42, cache_prompt false and the flag for no thinking (enable_thinking false). Every speed request carries a nonce, and the cache check of the tool found no cached tokens in any of the 63 requests that it checked.

Foreign load. The run's guard dropped and repeated two steps (requests at once, 4 at once, round 1 and context depth 64K, code prompt, run 1) because a foreign job appeared on the box; the discarded attempts are not published and the repeats ran after 35 s and 23 s of quiet. That is 2 of 40 steps; no step was invalidated by outside load. One more observation, in the step with 8 requests at once (round 2): in the last seconds of that step one foreign process used 280 % of a core in a single sample of 0.5 s. The guard repeats a step when one foreign process stays above 25 % for two samples in a row or when a foreign job shows up (the two repeats above were caused by a foreign file-copy job, as the published log shows); a single sample does not trigger it, so the step was kept. The note in the published raw files that lists a foreign CPU load of up to 88 % names a Python process of the run's own environment, at a moment when the card was 95 % busy, which points to the engine's own process and not to an outside job. The total of that round is 67.83 tok/s against 68.67 tok/s in round 1 of the same level. Not published: the guard's sample file, the quiet-check tables and the single requests of the speed run.

Against the first page of this setup. The first page (one stream, window 16,384 tokens, same speed test, medians from its raw file) measured: decode code 105.8 tok/s, decode prose 90.1 tok/s, 62.3 words per second, prefill 910.1 tok/s at 2K and 905.4 tok/s at 8K. This run: decode code 108.8, decode prose 83.7, 63.1 words per second, prefill 906.8 at 2K and 902.6 at 8K. No other number is compared here, because the test steps differ.

Not explained: in the quiet check before the server was started (a file that is not published), the second card reported full utilisation with no compute process on it; the same check before the first page's run did not. Prefill is 0.4 % at 2K and 0.3 % at 8K below the first page's values. We did not find the cause and did not measure again.

What we checked about the outputs. The same 104,101-token prompt gave the same answer on three separate server starts of the patched engine (probes, not register runs; the files of these probes are not published). Not shown: an equality check against a single-card run (the model does not fit one 16 GB card) or against the unpatched engine on this window (the unpatched engine does not start on these two 16 GB cards with this reserve setting; this was seen in probes whose files are not published).

python3 speed2.py --base-url http://127.0.0.1:43631/v1 --label tp2-py066-ownpatch2-w109k-1211-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/tp2-py066-ownpatch2-w109k-1211-speed-r0/tp2-py066-ownpatch2-w109k-1211-speed-r0/speed2 --contract $RUN/tp2-py066-ownpatch2-w109k-1211-speed-r0/tp2-py066-ownpatch2-w109k-1211-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,16384,32768,65536 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/pack-qwen38-27b-mlx4bit/tokenizer.json --tokenizer-python $BENCH/venv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253 (same as the other Qwen3.8-27B cards); the server default (checkpoint generation_config.json: temperature 1.0, top_p 0.95, top_k 20) is overridden per request, which the API allows (docs/api.md at v0.6.6, lines 79 and 105). Ash's command sets no sampling flag. Seed 42 for speed/quality. --model bench --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source TensorFold v0.6.6 src/tensorfold/cli.py:267 (--parallel auto = one stream on CUDA); the recipe command sets no --parallel; startup log has no 'streams' line --server-ctx 109511 --server-ctx-source engine startup line with our own patch and TENSORFOLD_MEMORY_RESERVE_GIB=2: 'CUDA rank 0 startup estimate 13.13 GiB within 13.13 GiB; native 262144, allocated prompt/reply window 109511' and 'context: 109511' in the serving line; the recipe command sets no --context. Checked in use: 104101- and 108810-token prompts answered, peak 13.6 GiB per card (expandable segments), no out-of-memory.

Quality

Not run: no Quality v2 test (MMLU-Pro, GPQA, IFEval, LiveCodeBench, long text) was run on this setup, so the page shows no quality number.

Also made this possible

• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• Model files (MLX 4-bit, revision 22d8d538); Hugging Face organisation TensorFold, whose only listed member is ashxhartTensorFoldHugging Face @ashxhart GitHub Website
• Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)Inco AI@inco_ai Hugging Face GitHub Website
• Speculative decoding method (DFlash) and the mirror we loadedz-lab@zhijianliu_ GitHub Hugging Face

Exact command

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

2x NVIDIA GeForce RTX 5060 Ti 16 GB (32 GB in total), one machine, tensor parallel over both cards (--tp 2). Driver 595.71.05, kernel 6.12.86+deb13-amd64

Model

Qwen/Qwen3.8-27B (dense 27B, Gated DeltaNet + attention), TensorFold/Qwen3.8-27B-MLX-4bit @22d8d538 (MLX affine 4-bit, group 64) + z-lab/Qwen3.8-27B-DFlash2 @50307d4c drafter

Model files (MLX 4-bit, revision 22d8d538); Hugging Face organisation TensorFold, whose only listed member is ashxhart

huggingface.co/TensorFold/Qwen3.8-27B-MLX-4bit

Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)

huggingface.co/z-lab/Qwen3.8-27B-DFlash2

Engine

Own patch on the frozen Python engine v0.6.6, not an official TensorFold release (Ash's TensorFold Python engine at v0.6.6, frozen); MLX 4-bit checkpoint and DFlash2 drafter; tensor parallel 2 over two cards; window 109,511 tokens instead of the recipe's native 262,144; 1 stream — the program that loads the model and answers requests

Recipe (docs/recipes/qwen3.8-27b.md at v0.6.6) and the frozen Python engine v0.6.6

github.com/ashhart/TensorFold/tree/v0.6.6 by Ash Hart

Steps

1. Get the recipe and the model files from the list above. The list has no link for the engine (Own patch on the frozen Python engine v0.6.6, not an official TensorFold release (Ash's TensorFold Python engine at v0.6.6, frozen); MLX 4-bit checkpoint and DFlash2 drafter; tensor parallel 2 over two cards; window 109,511 tokens instead of the recipe's native 262,144; 1 stream). The stock v0.6.6 engine without the own patch is not this setup.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8080, only from this computer.

   Copy

   env NCCL_DEBUG=WARN \
     TENSORFOLD_MEMORY_RESERVE_GIB=2 \
     PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
     CUDA_VISIBLE_DEVICES=<GPU1> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit \
     --tp 2 \
     --rank 1 \
     --master 127.0.0.1 & sleep 3 ; env NCCL_DEBUG=WARN \
     TENSORFOLD_MEMORY_RESERVE_GIB=2 \
     PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
     CUDA_VISIBLE_DEVICES=<GPU0> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit \
     --tp 2 \
     --rank 0 \
     --master 127.0.0.1 \
     --name bench \
     --host 127.0.0.1 \
     --port 8080

   What the parts mean

   • CUDA_VISIBLE_DEVICES=<GPU1>which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints
   • --host 127.0.0.1who can connect: only this computer
   • --port 8080the network port the server listens on
   • --name benchthe model ID the server advertises to clients

Own patch on the frozen Python engine v0.6.6, not an official TensorFold release (TensorFold python engine 0.6.6, tag cb2ebf0, with our own patch: each rank loads only its slice of the weights, and the start estimate of the window for two ranks counts the attention cache the way the engine already counts it on one card, so the window grows from 16,384 to 109,511 tokens; rank 1 also frees the buffers of the last request before the next one is allocated), qwen3_5 CUDA family, 2 x RTX 5060 Ti tensor parallel 2, DFlash2 drafter, one stream, window 109511; env NCCL_DEBUG=WARN TENSORFOLD_MEMORY_RESERVE_GIB=2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=<GPU1> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit --tp 2 --rank 1 --master 127.0.0.1 & sleep 3 ; env NCCL_DEBUG=WARN TENSORFOLD_MEMORY_RESERVE_GIB=2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=<GPU0> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit --tp 2 --rank 0 --master 127.0.0.1 --name bench --host 127.0.0.1 --port 8080 (rank 1 first, then rank 0; recipe docs/recipes/qwen3.8-27b.md at v0.6.6, lines 135 to 142; Own patch on the frozen Python engine v0.6.6, not an official TensorFold release; the own patch is not published)