Qwen3.8-27B UD-IQ3_S on one RTX 5060 Ti 16 GB: llama.cpp b12359 with Unsloth's MTP head

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-14. Raw files: speed result · speed contract · log · long answer.

Requests at once: the server has 1 slot. C2, C4 and C8 exceed it: the extra requests wait in the queue, and they were measured anyway. What ran. llama.cpp in the prebuilt Linux package b12359 (CUDA 13.5, commit b1b770d90) of ggml-org. This is the build that the v1.0.0 release names: the release page has no binaries, only a file that names b12359; the v1.0.0 tag points to the same commit, b1b770d90. It served Qwen3.8-27B in Unsloth's 3-bit GGUF file UD-IQ3_S (repository revision 4af51b35; the file itself was last changed on 19 August 2026 according to its history on Hugging Face, checked on 15 November, so before Unsloth's Dynamic v3.1 update of 11 November), with the MTP head from the same repository (MTP/mtp-Qwen3.8-27B-Q4_0.gguf), started with --spec-type draft-mtp --spec-draft-n-max 2, as Unsloth's MTP guide says. Where this run differs from the author's way. (1) File: Unsloth's page names UD-Q3_K_XL for 3-bit. With it, the server's own memory fit left a window of only 17,408 tokens on this card (with -np 1), so we took UD-IQ3_S, which gave 32,768 tokens. (2) Slots: -np 1 instead of the automatic setting. With the automatic setting the server started 4 slots that share a pool of only 13,056 tokens; -np 1 gave one slot with 32,768 tokens. (3) Window: -c 36864 is ours. The server's own choice of 32,768 tokens cannot take a 32,768-token prompt plus the 512 output tokens of the speed test, and a first job with that window (llama10-5060-iq3s-mtp-1311, speed only) had no value at the 32K step. The window of this run is 36,864 tokens, so the 64K and 128K steps do not fit and have no value. (4) --fit off -ngl all: this one changes the author's default, which is -fit on (tools/server/README.md of b12359). With -c 36864 and -fit on, a test run decoded clearly slower (36.8 tokens per second for code and 33.6 for prose, against 52.4 and 47.9 here), and the GPU memory in use after loading was 14,892 MiB, against 15,142 MiB with --fit off -ngl all; the server log does not say what -fit placed where, but this pattern fits layers moved to the CPU. With --fit off -ngl all every layer is on the card (the card reports 16,311 MiB in total). That test run is not part of the register and its raw files are not on this page. (5) The vision projector is not loaded (--no-mmproj), for memory. (6) The model and the MTP head come from local files instead of -hf, and --cache-ram 0, --no-webui and --metrics are ours. The code decode value varied more between the runs of this test than the other values (13.0 % between the runs); the page shows the spread. We found no decode speed published by ggml-org or Unsloth for this model on an RTX 5060 Ti or any 16 GB card (checked on 13 November 2026), so there is no figure of the authors next to ours.

python3 speed2.py --base-url http://127.0.0.1:45939/v1 --label llama10-5060-iq3s-mtp-c36864-fitoff-1311-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/llama10-5060-iq3s-mtp-c36864-fitoff-1311-speed-r0/llama10-5060-iq3s-mtp-c36864-fitoff-1311-speed-r0/speed2 --contract $RUN/llama10-5060-iq3s-mtp-c36864-fitoff-1311-speed-r0/llama10-5060-iq3s-mtp-c36864-fitoff-1311-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source unsloth/Qwen3.8-27B-GGUF README.md:83-84 @4af51b35 (Thinking 1.0/0.95/20/0.0/presence 0.0; Instruct 0.7/0.8/20/0.0/presence 1.5, repetition_penalty 1.0), same as Qwen/Qwen3.8-27B; our parts: thinking off = Instruct profile --model qwen38-27b-llamacpp --no-cache-prompt --server-log $HOME/llama10-run/logs/server.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source FORCED by memory, named deviation from the author default (-np auto, tools_server_README.md:178): with -np auto the server starts 4 slots that share a unified KV pool of only 13,056 tokens (probe Nov 13, 2026 'n_slots = 4, n_ctx_slot = 13056'); -np 1 gives 'n_slots = 1, n_ctx_slot = 32768'. Same step as ByteShape's author table (-np 1). --server-ctx 36864 --server-ctx-source Named deviation from the author default (-c 0 with -fit on gave 32768 with -np 1, which cannot take a 32768-token prompt plus 512 output tokens: first job llama10-5060-iq3s-mtp-1311, 'PREFILL d32768 FAILED ... exceeds the available context size (32768)'): -c 36864 set by us so the 32K depths fit; model and flags otherwise unchanged. Memory: the probe at 32768 used 14,862 MiB of 16,311 MiB. Variant fitoff: with -c given and -fit on, the first c36864 job decoded only 36.8/33.6 tok/s against 47.2/46.4 at the fit-chosen 32768 window (GPU memory only +30 MiB over the 32768 start: -fit most likely moved layers to the CPU), so --fit off -ngl all forces all layers on the card.

How well it answers

Thinking off (enable_thinking false in every request), the instruct sampling of the model card (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5), seed 42, 1 request at a time, the same server start as the speed test (window 36,864, 1 slot, --fit off). Answer limits: 48 tokens for MMLU-Pro and GPQA (one letter), 4,096 for IFEval, 128 for the long-text tasks (RULER). A cut-off answer counts as wrong: in the long-text test 25 of 80 answers were cut off at the limit of 128 tokens (14 of 40 at 8K, 11 of 40 at 32K), so the two long-text numbers also measure that limit. 2 MMLU-Pro answers and 3 GPQA answers had no usable letter and count as wrong. Raw numbers only, no overall score. GPQA ran on the public mirror fingertap/GPQA-Diamond (198 questions), because the original needs a login.

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-14.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-ud-iq3s-mtp-llamacpp-b12359-5060ti-1311-c36864-fitoff.json --parts mmlupro --job llama10-5060-iq3s-mtp-c36864-fitoff-1311

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-ud-iq3s-mtp-llamacpp-b12359-5060ti-1311-c36864-fitoff.json --parts gpqa --job llama10-5060-iq3s-mtp-c36864-fitoff-1311

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-ud-iq3s-mtp-llamacpp-b12359-5060ti-1311-c36864-fitoff.json --parts ifeval --job llama10-5060-iq3s-mtp-c36864-fitoff-1311

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-ud-iq3s-mtp-llamacpp-b12359-5060ti-1311-c36864-fitoff.json --parts ruler --job llama10-5060-iq3s-mtp-c36864-fitoff-1311

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-ud-iq3s-mtp-llamacpp-b12359-5060ti-1311-c36864-fitoff.json --parts ruler --job llama10-5060-iq3s-mtp-c36864-fitoff-1311

Also made this possible

• Model files on Hugging FaceUnsloth AI@unslothai Hugging Face GitHub Website
• Engine (llama.cpp, prebuilt package b12359)ggml-org (llama.cpp)@ggml_org GitHub Hugging Face Website
• Model (Qwen3.8-27B)Qwen@Alibaba_Qwen Hugging Face GitHub Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5060 Ti 16 GB (one machine, second card not used), driver 595.71.05, kernel 6.12.86 (Debian 13)

Model

Qwen3.8-27B UD-IQ3_S GGUF (unsloth/Qwen3.8-27B-GGUF @4af51b35) + MTP head Q4_0 (same repo)

Model files on Hugging Face

huggingface.co/unsloth/Qwen3.8-27B-GGUF

Model (Qwen3.8-27B)

huggingface.co/Qwen/Qwen3.8-27B

Engine

ggml-org's llama.cpp package b12359 with Unsloth's 3-bit GGUF and its MTP head (draft-mtp), window 36,864 tokens — the program that loads the model and answers requests · github.com/ggml-org/llama.cpp/releases/tag/b12359

Recipe (MTP guide: --spec-type draft-mtp --spec-draft-n-max 2) and model files (UD-IQ3_S and MTP head, repository revision 4af51b35)

unsloth.ai/docs/models/mtp by Unsloth AI

Steps

1. Get the recipe, the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   llama-server \
     -m Qwen3.8-27B-UD-IQ3_S.gguf \
     --no-mmproj \
     --spec-type draft-mtp \
     -md mtp-Qwen3.8-27B-Q4_0.gguf \
     --spec-draft-n-max 2 \
     -np 1 \
     -c 36864 \
     --fit off \
     -ngl all \
     --alias qwen38-27b-llamacpp \
     --host 127.0.0.1 \
     --port 8888 \
     --cache-ram 0 \
     --no-webui \
     --metrics

   What the parts mean

   • -m Qwen3.8-27B-UD-IQ3_S.ggufthe model file it loads
   • -c 36864the context window in tokens: how much text the model can see at once
   • -np 1how many requests it answers at the same time
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on
   • --cache-ram 0the most system RAM, in MiB, used to keep earlier prompts for reuse (0 turns this off)
   • --no-webuiturns off the built-in chat web page; only the API answers
   • --metricsturns on a /metrics page with server statistics (Prometheus format)
   • --spec-…settings for speculative decoding: a draft guesses the next tokens and the model checks them (--spec-type: the kind of draft, --spec-draft-n-max: the most tokens drafted per step)
   • --alias qwen38-27b-llamacppthe model name clients use in their requests

llama.cpp b12359 @b1b770d90 (prebuilt package ubuntu-cuda-13.5-x64 of ggml-org, the build that the v1.0.0 release names; the v1.0.0 tag points to the same commit), MTP draft-mtp n-max 2, f16 KV; llama-server -m Qwen3.8-27B-UD-IQ3_S.gguf --no-mmproj --spec-type draft-mtp -md mtp-Qwen3.8-27B-Q4_0.gguf --spec-draft-n-max 2 -np 1 -c 36864 --fit off -ngl all --alias qwen38-27b-llamacpp --host 127.0.0.1 --port 8888 --cache-ram 0 --no-webui --metrics (Unsloth's MTP guide: --spec-type draft-mtp --spec-draft-n-max 2, nothing else; our changes to the author's way are -np 1, -c 36864, --fit off -ngl all, --no-mmproj and the file UD-IQ3_S, with the reasons under the speed results; local files instead of -hf; --cache-ram 0, --no-webui, --metrics and the alias are ours)