Qwen3.8-27B on two RTX 5060 Ti: Ash's frozen Python engine v0.6.6 with our own patch, tensor parallel 2

How fast it runs

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-12. Raw files: speed result · speed contract · log · long answer (empty file, no valid result).

What ran. Own patch on the frozen Python engine v0.6.6, not an official TensorFold release: Ash's TensorFold Python engine at its tag v0.6.6 (commit cb2ebf0), the version that the recipe names; the engine is frozen at that version. The own patch does one thing: each rank loads only its slice of the weights. It is not published and it is not part of any TensorFold release. The model is Qwen3.8-27B (a dense model) in the MLX 4-bit checkpoint TensorFold/Qwen3.8-27B-MLX-4bit (revision 22d8d538), with the DFlash2 drafter z-lab/Qwen3.8-27B-DFlash2 (revision 50307d4c), served on two RTX 5060 Ti with tensor parallel 2: rank 1 is started first, then rank 0, as the recipe says (docs/recipes/qwen3.8-27b.md at v0.6.6, lines 135 to 142). The software is torch 2.14.1 for CUDA 13 and Python 3.13.5, with driver 595.71.05.

Where this run differs from the recipe. (1) Engine: the own patch on the frozen v0.6.6 (see above), not the author's code. (2) Window: 16,384 tokens (prompt and reply together), where the recipe's native window is 262,144. The recipe command sets no --context; the engine's own start estimate gives the window. With TENSORFOLD_MEMORY_RESERVE_GIB=2, the memory reserve that the RUNBOOK of v0.6.6 names (line 117) and that the recipe command does not set, the start line reads 'CUDA rank 0 startup estimate 11.55 GiB within 13.13 GiB; native 262144, allocated prompt/reply window 16384'. The window limit comes from the engine's start-up estimate, not from the card being full: about 10 GiB per card right after loading (9,968 MiB), card 0 at 12,386 MiB under load (the run's own memory readings, not part of the published raw files). (3) Streams: one. The engine serves one request at a time on CUDA (TensorFold v0.6.6, src/tensorfold/cli.py line 267 and cli_args.py lines 104 to 107), and the recipe command sets no --parallel. (4) Sampling: the speed test sends the instruct profile of the model card (temperature 0.7, top_p 0.8, top_k 20, min_p 0, presence_penalty 1.5) with every request and turns thinking off; the server's own default shown in its start line (temperature 1.0, top_k 20, top_p 0.95) is overridden per request, which the API allows. The recipe command sets no sampling flag. (5) The drafts are on (the start line reads 'drafts: on'). (6) Addresses: both ranks run on one machine, so master and host are 127.0.0.1 (the recipe shows 192.0.2.1 and 0.0.0.0).

Window and memory. The window is 16,384 tokens and the server runs 1 stream; the start line records 'context: 16384' and 'loaded in 241.3s'. The engine serves one request at a time, so the levels 2, 4 and 8 at once are above its slot count.

Requests at once. The server has 1 stream: the requests wait in a queue and run one after another, not in parallel. The total stays near the value of a single request (71.2, 65.8, 69.4 and 67.6 tok/s with 1, 2, 4 and 8 requests), and the speed of each request falls as the queue grows (71.2, 49.7, 28.9 and 15.3 tok/s). All requests carry the same prompt.

Dropped steps (reason: the server window of 16,384 tokens): the prefill step at 32K was left out of the command because it cannot fit; the context steps at 16K, 32K, 64K and 128K and the long-answer step have no value. Prefill and context from 16K up: does not fit. The tool checked each of the four context sizes before it sent anything: a prompt of 16K, 32K, 64K or 128K tokens plus the 512 reply tokens is larger than the 16,384-token window that the server reports (16,384 plus 512 is 16,896, already above it), so no request was sent for those sizes and they have no value (the register rows speed2_ctx_d16384_fits, speed2_ctx_d32768_fits, speed2_ctx_d65536_fits and speed2_ctx_d131072_fits are 0). The prefill series has two sizes, 2K and 8K (910 and 905 tok/s, first token after 2.25 s and 9.05 s); the 8K prefill is the deepest prompt that was measured.

Long answer test: no result. The test asks for one reply of up to 32,768 tokens, which is more than the whole window of 16,384 tokens: the server refused the request at once (HTTP 400, the request asks for 32,768 reply tokens against a window of 16,384). The output file l1-output.txt is empty. The step is marked invalid, has no value and is not in the register.

The headline 98.0 tok/s is the mean of the code prompt (105.8) and the prose prompt (90.1), medians of two runs each (spread (CV) of 1.0 % and 1.5 %). The code prompt has 1,254 and 1,259 tokens and the prose prompt 1,349 and 1,350 tokens, and every reply is 512 tokens long and ends at the limit. Acceptance rate and mean accepted length: not available (the engine does not report them, so the test counted no drafted tokens).

Sampling. The speed test sent temperature 0.7, top_p 0.8, top_k 20, min_p 0 and presence_penalty 1.5 with every request (the instruct profile of the model card), seed 42, cache_prompt false and the flag for no thinking (enable_thinking false). Every speed request carries a nonce, and the cache check of the tool found no cached tokens in any of the 41 requests that it checked.

Foreign load. The run's guard dropped and repeated one step (decode, code prompt, run 1) because outside load appeared on the box; the discarded attempt is not published and the repeat ran after 35 s of quiet. That is 1 of 19 steps; no step was invalidated by outside load. Not published: the guard's sample file, the quiet-check tables and the single requests of the speed run.

python3 speed2.py --base-url http://127.0.0.1:33391/v1 --label tp2-py066-ownpatch-1211-r2-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/tp2-py066-ownpatch-1211-r2-speed-r0/tp2-py066-ownpatch-1211-r2-speed-r0/speed2 --contract $RUN/tp2-py066-ownpatch-1211-r2-speed-r0/tp2-py066-ownpatch-1211-r2-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/pack-qwen38-27b-mlx4bit/tokenizer.json --tokenizer-python $BENCH/venv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253 (same as the other Qwen3.8-27B cards); the server default (checkpoint generation_config.json: temperature 1.0, top_p 0.95, top_k 20) is overridden per request, which the API allows (docs/api.md at v0.6.6, lines 79 and 105). Ash's command sets no sampling flag. Seed 42 for speed/quality. --model bench --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source TensorFold v0.6.6 src/tensorfold/cli.py:267 (--parallel auto = one stream on CUDA) and cli_args.py:104-107 ('CUDA: one at a time, the others waiting their turn'); the recipe command sets no --parallel; startup log has no 'streams' line --server-ctx 16384 --server-ctx-source engine startup line at the author's command with TENSORFOLD_MEMORY_RESERVE_GIB=2 (RUNBOOK.md:117): 'CUDA rank 0 startup estimate 11.55 GiB within 13.13 GiB; native 262144, allocated prompt/reply window 16384' and 'context: 16384' in the serving line; the recipe command sets no --context

Quality

Not run: no Quality v2 test (MMLU-Pro, GPQA, IFEval, LiveCodeBench, long text) was run on this setup, so the page shows no quality number.

Also made this possible

• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• Model files (MLX 4-bit, revision 22d8d538); Hugging Face organisation TensorFold, whose only listed member is ashxhartTensorFoldHugging Face @ashxhart GitHub Website
• Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)Inco AI@inco_ai Hugging Face GitHub Website
• Speculative decoding method (DFlash) and the mirror we loadedz-lab@zhijianliu_ GitHub Hugging Face

Exact command

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

2x NVIDIA GeForce RTX 5060 Ti 16 GB (32 GB in total), one machine, tensor parallel over both cards (--tp 2). Driver 595.71.05, kernel 6.12.86+deb13-amd64

Model

Qwen/Qwen3.8-27B (dense 27B, Gated DeltaNet + attention), TensorFold/Qwen3.8-27B-MLX-4bit @22d8d538 (MLX affine 4-bit, group 64) + z-lab/Qwen3.8-27B-DFlash2 @50307d4c drafter

Model files (MLX 4-bit, revision 22d8d538); Hugging Face organisation TensorFold, whose only listed member is ashxhart

huggingface.co/TensorFold/Qwen3.8-27B-MLX-4bit

Draft model (DFlash2, revision 50307d4c; the repository we loaded is z-lab's mirror of incoai/Qwen3.8-27B-DFlash2)

huggingface.co/z-lab/Qwen3.8-27B-DFlash2

Engine

Own patch on the frozen Python engine v0.6.6, not an official TensorFold release (Ash's TensorFold Python engine at v0.6.6, frozen); MLX 4-bit checkpoint and DFlash2 drafter; tensor parallel 2 over two cards; window 16,384 tokens instead of the recipe's native 262,144; 1 stream — the program that loads the model and answers requests

Recipe (docs/recipes/qwen3.8-27b.md at v0.6.6) and the frozen Python engine v0.6.6

github.com/ashhart/TensorFold/tree/v0.6.6 by Ash Hart

Steps

1. Get the recipe and the model files from the list above. The list has no link for the engine (Own patch on the frozen Python engine v0.6.6, not an official TensorFold release (Ash's TensorFold Python engine at v0.6.6, frozen); MLX 4-bit checkpoint and DFlash2 drafter; tensor parallel 2 over two cards; window 16,384 tokens instead of the recipe's native 262,144; 1 stream). The stock v0.6.6 engine without the own patch is not this setup.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8080, only from this computer.

   Copy

   env NCCL_DEBUG=WARN \
     TENSORFOLD_MEMORY_RESERVE_GIB=2 \
     CUDA_VISIBLE_DEVICES=<GPU1> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit \
     --tp 2 \
     --rank 1 \
     --master 127.0.0.1 & sleep 3 ; env NCCL_DEBUG=WARN \
     TENSORFOLD_MEMORY_RESERVE_GIB=2 \
     CUDA_VISIBLE_DEVICES=<GPU0> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit \
     --tp 2 \
     --rank 0 \
     --master 127.0.0.1 \
     --name bench \
     --host 127.0.0.1 \
     --port 8080

   What the parts mean

   • CUDA_VISIBLE_DEVICES=<GPU1>which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints
   • --host 127.0.0.1who can connect: only this computer
   • --port 8080the network port the server listens on
   • --name benchthe model ID the server advertises to clients

Own patch on the frozen Python engine v0.6.6, not an official TensorFold release (TensorFold python engine 0.6.6, tag cb2ebf0, with our own patch: each rank loads only its slice of the weights), qwen3_5 CUDA family, 2 x RTX 5060 Ti tensor parallel 2, DFlash2 drafter, one stream, window 16384; env NCCL_DEBUG=WARN TENSORFOLD_MEMORY_RESERVE_GIB=2 CUDA_VISIBLE_DEVICES=<GPU1> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit --tp 2 --rank 1 --master 127.0.0.1 & sleep 3 ; env NCCL_DEBUG=WARN TENSORFOLD_MEMORY_RESERVE_GIB=2 CUDA_VISIBLE_DEVICES=<GPU0> tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit --tp 2 --rank 0 --master 127.0.0.1 --name bench --host 127.0.0.1 --port 8080 (rank 1 first, then rank 0; recipe docs/recipes/qwen3.8-27b.md at v0.6.6, lines 135 to 142; Own patch on the frozen Python engine v0.6.6, not an official TensorFold release; the own patch is not published)