Qwen3.8-27B on two RTX 5060 Ti: Ash's frozen Python engine v0.6.6 with our own patch, four streams at once, tensor parallel 2

How fast it runs

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer (empty file, no valid result).

What ran. Own patch on the frozen Python engine v0.6.6, not an official TensorFold release: Ash's TensorFold Python engine at its tag v0.6.6 (commit cb2ebf0), the version whose recipe we followed; the engine is frozen at that version. The own patch is ours, not Ash's. It changes: each rank loads only its slice of the weights; the same build also corrects the window estimate for one stream (see the page of the 109,511-token window), but for several streams the window per stream comes from the engine's own estimate, which the patch does not change. It is not published and it is not part of any TensorFold release. The model is Qwen3.8-27B (a dense model) in the MLX 4-bit checkpoint TensorFold/Qwen3.8-27B-MLX-4bit (revision 22d8d538), with the DFlash2 drafter z-lab/Qwen3.8-27B-DFlash2 (revision 50307d4c), served on two RTX 5060 Ti with tensor parallel 2: rank 1 is started first, then rank 0, as the recipe says (docs/recipes/qwen3.8-27b.md at v0.6.6, lines 135 to 142). The software is torch 2.14.1 for CUDA 13 and Python 3.13.5, with driver 595.71.05.

Where this run differs from the recipe. (1) Engine: the own patch on the frozen v0.6.6 (see above), not the author's code. (2) Window: 10,549 tokens per stream (prompt and reply together), where the recipe's native window is 262,144. Each of the four streams reserves its own window, so the window per stream is smaller than with one stream. The recipe command sets no --context; the engine's own start estimate gives the window. With TENSORFOLD_MEMORY_RESERVE_GIB=2 (RUNBOOK of v0.6.6 line 117; docs/recipes/cuda.md lines 206 to 208), the start line reads 'CUDA rank 0 startup estimate 13.13 GiB within 13.13 GiB; native 262144, allocated prompt/reply window 10549, cache slots 10550'. The patch does not change the engine's estimate for several streams. (3) Streams: four (--parallel 4 --checkpoint-slots 1). --parallel is documented for the 27B on CUDA (docs/recipes/cuda.md line 197, RUNBOOK.md lines 73 to 75: the same number on both ranks) and --checkpoint-slots in README.md line 135; the recipe's command for two ranks sets neither, and its example for concurrent requests (docs/recipes/qwen3.8-27b.md lines 149 to 153) uses --parallel 16 --context 8192. We set no --context. (4) Sampling: the speed test sends the instruct profile of the model card (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence_penalty 1.5) with every request and turns thinking off; the server's own default shown in its start line (temperature 1.0, top_k 20, top_p 0.95) is overridden per request, which the API allows. The recipe command sets no sampling flag. (5) The drafts are on (the start line reads 'drafts: on'). (6) Environment: TENSORFOLD_MEMORY_RESERVE_GIB=2 (see (2)) and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which docs/recipes/cuda.md (lines 209 and 210) names as a way to reduce fragmentation near the memory cap; the recipe command sets neither. (7) Addresses: both ranks run on one machine, so master and host are 127.0.0.1 (the recipe shows 192.0.2.1 and 0.0.0.0).

Window and streams. The window is 10,549 tokens per stream and the server runs 4 streams (--parallel 4 --checkpoint-slots 1); the start line records 'context: 10549' and 'loaded in 236.3s'. The start line of the measured run that names the streams ('4 streams of 10549 prompt/reply tokens') is quoted in the notes of the speed test in the published raw files; the start log of the measured run itself is not published, so we read the line in the start log of a probe with the same flags. Level 8 at once is above the stream count.

Memory. Right after loading, each card holds 9,010 MiB. In the run itself, card 0 showed up to 10,540 MiB (the largest of the readings taken about every 10 seconds). None of these memory readings is part of the published raw files.

Requests at once. The server has 4 streams: up to four requests decode together in shared rounds. The total rises with the number of requests until the streams are full (67.4, 99.0, 119.3 and 120.2 tok/s with 1, 2, 4 and 8 requests), and the speed of each request falls (67.4, 49.9, 30.5 and 19.5 tok/s). With 8 requests at once, four run and four wait, so the per-request value includes the waiting. All requests carry the same prompt.

Draft acceptance. The engine's counters report 1,540 of 5,238 drafted tokens accepted (29.4 %) and a mean accepted length of 4.1 tokens. The single-stream page of this setup reports no acceptance numbers (its engine run did not report them).

Dropped steps (reason: the window of 10,549 tokens per stream): the context steps at 16K, 32K, 64K and 128K do not fit; the prefill series has the two sizes 2K and 8K (907 and 902 tok/s, first token after 2.26 s and 9.08 s). The tool checked each of the four context sizes before it sent anything: a prompt of 16K, 32K, 64K or 128K tokens plus the 512 reply tokens is larger than the 10,549-token window that the server reports (16,384 plus 512 is 16,896), so no request was sent for those sizes and they have no value (the register rows speed2_ctx_d16384_fits, speed2_ctx_d32768_fits, speed2_ctx_d65536_fits and speed2_ctx_d131072_fits are 0).

Long answer test: no result. The test asks for one reply of up to 32,768 tokens, which is more than the whole window of 10,549 tokens per stream: the server refused the request at once (HTTP 400, the request asks for 32,768 reply tokens against a window of 10,549). The output file l1-output.txt is empty. The step is marked invalid, has no value and is not in the register.

The headline 93.9 tok/s is the mean of the code prompt (97.2) and the prose prompt (90.6), medians of two runs each (spread (CV) of 7.5 % and 9.7 %). The code prompt has 1,258 and 1,261 tokens and the prose prompt 1,354 and 1,354 tokens, and every reply is 512 tokens long and ends at the limit.

Sampling. The speed test sent temperature 0.7, top_p 0.8, top_k 20, min_p 0 and presence_penalty 1.5 with every request (the instruct profile of the model card), seed 42, cache_prompt false and the flag for no thinking (enable_thinking false). Every speed request carries a nonce, and the cache check of the tool found no cached tokens in any of the 41 requests that it checked.

Foreign load. The load guard of the speed test (any other process above 25 % of a core for two samples in a row, or a foreign job) threw away and repeated no step: 0 of 19 steps. The note in the published raw files that lists a foreign CPU load of up to 50 % names a Python process of the run's own environment, at a moment when the card was 92 % busy, which points to the engine's own process and not to an outside job. Not published: the guard's sample file, the quiet-check tables and the single requests of the speed run.

Against the first page of this setup. The first page (one stream, window 16,384 tokens, same speed test, medians from its raw file) measured: decode code 105.8 tok/s, decode prose 90.1 tok/s, 62.3 words per second, prefill 910.1 tok/s at 2K and 905.4 tok/s at 8K. This run: decode code 97.2, decode prose 90.6, 64.3 words per second, prefill 906.7 at 2K and 902.0 at 8K. No other number is compared here, because the test steps differ.

Not explained: in the quiet check before the server was started (a file that is not published), the second card reported full utilisation with no compute process on it; the same check before the first page's run did not. Prefill is 0.4 % at both 2K and 8K below the first page's values. We did not find the cause and did not measure again.

python3 speed2.py --base-url http://127.0.0.1:43951/v1 --label tp2-py066-ownpatch2-par4-1211-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/tp2-py066-ownpatch2-par4-1211-speed-r0/tp2-py066-ownpatch2-par4-1211-speed-r0/speed2 --contract $RUN/tp2-py066-ownpatch2-par4-1211-speed-r0/tp2-py066-ownpatch2-par4-1211-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/pack-qwen38-27b-mlx4bit/tokenizer.json --tokenizer-python $BENCH/venv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253 (same as the other Qwen3.8-27B cards); the server default (checkpoint generation_config.json: temperature 1.0, top_p 0.95, top_k 20) is overridden per request, which the API allows (docs/api.md at v0.6.6, lines 79 and 105). Ash's command sets no sampling flag. Seed 42 for speed/quality. --model bench --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 4 --server-slots-source recipe docs/recipes/qwen3.8-27b.md:152-155 and docs/recipes/cuda.md:197 (--parallel N decodes up to N requests in shared rounds; pass the same N on both ranks, RUNBOOK.md:73-75); startup line '4 streams of 10549 prompt/reply tokens, 1 prompt states kept' --server-ctx 10549 --server-ctx-source engine startup line with our own patch, --parallel 4 --checkpoint-slots 1 and TENSORFOLD_MEMORY_RESERVE_GIB=2: 'CUDA rank 0 startup estimate 13.13 GiB within 13.13 GiB; native 262144, allocated prompt/reply window 10549' and '4 streams of 10549 prompt/reply tokens, 1 prompt states kept' (this line is from the start log of a probe with the same flags; the start log of the measured run is not published); the recipe command sets no --context

Quality

Not run: no Quality v2 test (MMLU-Pro, GPQA, IFEval, LiveCodeBench, long text) was run on this setup, so the page shows no quality number.

Also made this possible

• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• Model files (MLX 4-bit, revision 22d8d538); Hugging Face organisation TensorFold, whose only listed member is ashxhartTensorFoldHugging Face @ashxhart GitHub Website
• Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)Inco AI@inco_ai Hugging Face GitHub Website
• Speculative decoding method (DFlash) and the mirror we loadedz-lab@zhijianliu_ GitHub Hugging Face

Exact command

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

2x NVIDIA GeForce RTX 5060 Ti 16 GB (32 GB in total), one machine, tensor parallel over both cards (--tp 2). Driver 595.71.05, kernel 6.12.86+deb13-amd64

Model

Qwen/Qwen3.8-27B (dense 27B, Gated DeltaNet + attention), TensorFold/Qwen3.8-27B-MLX-4bit @22d8d538 (MLX affine 4-bit, group 64) + z-lab/Qwen3.8-27B-DFlash2 @50307d4c drafter

Model files (MLX 4-bit, revision 22d8d538); Hugging Face organisation TensorFold, whose only listed member is ashxhart

huggingface.co/TensorFold/Qwen3.8-27B-MLX-4bit

Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)

huggingface.co/z-lab/Qwen3.8-27B-DFlash2

Engine

Own patch on the frozen Python engine v0.6.6, not an official TensorFold release (Ash's TensorFold Python engine at v0.6.6, frozen); MLX 4-bit checkpoint and DFlash2 drafter; tensor parallel 2 over two cards; 4 streams (--parallel 4 --checkpoint-slots 1), window 10,549 tokens per stream instead of the recipe's native 262,144 — the program that loads the model and answers requests

Recipe (docs/recipes/qwen3.8-27b.md at v0.6.6) and the frozen Python engine v0.6.6

github.com/ashhart/TensorFold/tree/v0.6.6 by Ash Hart

Steps

1. Get the recipe and the model files from the list above. The list has no link for the engine (Own patch on the frozen Python engine v0.6.6, not an official TensorFold release (Ash's TensorFold Python engine at v0.6.6, frozen); MLX 4-bit checkpoint and DFlash2 drafter; tensor parallel 2 over two cards; 4 streams (--parallel 4 --checkpoint-slots 1), window 10,549 tokens per stream instead of the recipe's native 262,144). The stock v0.6.6 engine without the own patch is not this setup.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8080, only from this computer.

   Copy

   env NCCL_DEBUG=WARN \
     TENSORFOLD_MEMORY_RESERVE_GIB=2 \
     PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
     CUDA_VISIBLE_DEVICES=<GPU1> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit \
     --tp 2 \
     --rank 1 \
     --master 127.0.0.1 \
     --parallel 4 \
     --checkpoint-slots 1 & sleep 3 ; env NCCL_DEBUG=WARN \
     TENSORFOLD_MEMORY_RESERVE_GIB=2 \
     PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
     CUDA_VISIBLE_DEVICES=<GPU0> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit \
     --tp 2 \
     --rank 0 \
     --master 127.0.0.1 \
     --name bench \
     --host 127.0.0.1 \
     --port 8080 \
     --parallel 4 \
     --checkpoint-slots 1

   What the parts mean

   • CUDA_VISIBLE_DEVICES=<GPU1>which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints
   • --parallel 4how many requests it answers at the same time
   • --host 127.0.0.1who can connect: only this computer
   • --port 8080the network port the server listens on
   • --name benchthe model ID the server advertises to clients

Own patch on the frozen Python engine v0.6.6, not an official TensorFold release (TensorFold python engine 0.6.6, tag cb2ebf0, with our own patch: each rank loads only its slice of the weights; the same build also corrects the window estimate for one stream (see the page of the 109,511-token window), but for several streams the window per stream comes from the engine's own estimate, which the patch does not change), qwen3_5 CUDA family, 2 x RTX 5060 Ti tensor parallel 2, DFlash2 drafter, four streams (--parallel 4 --checkpoint-slots 1), window 10549 per stream; env NCCL_DEBUG=WARN TENSORFOLD_MEMORY_RESERVE_GIB=2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=<GPU1> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit --tp 2 --rank 1 --master 127.0.0.1 --parallel 4 --checkpoint-slots 1 & sleep 3 ; env NCCL_DEBUG=WARN TENSORFOLD_MEMORY_RESERVE_GIB=2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True CUDA_VISIBLE_DEVICES=<GPU0> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit --tp 2 --rank 0 --master 127.0.0.1 --name bench --host 127.0.0.1 --port 8080 --parallel 4 --checkpoint-slots 1 (rank 1 first, then rank 0; recipe docs/recipes/qwen3.8-27b.md at v0.6.6, lines 135 to 142; Own patch on the frozen Python engine v0.6.6, not an official TensorFold release; the own patch is not published)