ByteShape's Qwen3.8-27B on a 16 GB card: the GPU-5 quant (IQ4_XS, 3.84 bpw) as GGUF with llama.cpp and MTP drafts

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K and 32K tokens (the steps of 64K and 128K do not fit into the 40,960-token window and were not run; a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-09. Raw files: speed result · speed contract · log · long answer.

The speed test ran with the README of ByteShape's GGUF repository (revision 3fdfbd9b): Qwen3.8-27B-IQ4_XS-3.84bpw.gguf, the quant the author calls GPU-5 (13.08 GB, sha256 checked against the Hugging Face file listing), with the official llama.cpp release b12223 (CUDA 13.5 build for Linux, sha256 checked against the digest on GitHub; not built by me) and the MTP drafts of the README's quick start (--spec-type draft-mtp --spec-draft-n-max 3; the MTP head is inside the GGUF). I used llama-server with local files instead of -hf, and the instruct sampling of the model card with thinking off, a fixed seed and two fixed prompts (code and prose). The author measures with another engine build (b10430), a mix of prompts and thinking-mode sampling, so the author's figures and these are not directly comparable.

Differences from the author's recipe, all forced by the 16 GB card or by my tests: (1) No vision projector: --no-mmproj instead of --mmproj-auto. In my probe start the model with the projector did not load at a context of 32,768 (out of memory while allocating the projector's 888 MiB buffer). The README itself names --no-mmproj for text use. So this page measures text only; vision is not measured. (2) One slot (-np 1): with the automatic slot count that the author's command leaves to the server, my probe start (with the projector) failed too, out of memory while allocating 2,394 MiB for the recurrent-state cache of the slots; -np 1 -fa on is what the README names for its throughput table. (3) A context of 40,960 tokens instead of the 32,768 of the author's tool: more, not less. The long-answer test (32,768 tokens), the 32K long-text test and the 32K prefill need a window above 32,768. The steps of 64K and 128K do not fit into this window (prompt plus 512 output tokens must fit), so the table shows no value for them. Other flags that do not change the numbers: -ngl all with -fit off (everything on the GPU), --cache-ram 0, --no-webui --metrics, host and port. Not measured: the vision projector, the DFlash 2 drafts that the README also describes (text only, about 1.2 GB more memory) and the GPU-4 quant.

The headline 44.6 tok/s is the mean of the code prompt (46.1) and the prose prompt (43.1), medians of two runs each (spread 0.6 % and 4.0 %). The draft acceptance of the speed test is 49.3 % of the drafted tokens, 2.48 tokens per step on average. A first speed run of this setup had a spread between its code runs above my limit of 15 %, so the speed test was run again; this page shows only that repeat. Requests at once: the server has 1 slot (the README's throughput setting), so 2, 4 and 8 requests queue behind each other and were measured anyway. The total stays between 39.3 and 39.6 tok/s for 1 to 8 requests at once; the speed of one request falls from 39.3 to 9.0 tok/s (the 1-request figure includes the prompt time, so it is below the decode speed). The load guard (a sample every half second) repeated 0 of its 30 steps. It recorded foreign peaks above 25 % of one core in 4 steps (git at 110 %, 112 % and 140 %, and find at 62 %, all of them the box's own mirror jobs); none lasted for two samples in a row, so none counted as a disturbance (the guard repeats a step only then). The quiet check (one sample every 10 seconds) saw at most 110 % of one core, from the box's own indexing service. The box runs its own services, and the guard is set to ignore the ones named in the speed contract. The GPU reading of the test tool in this speed test is not that of the measured card (it stays at one constant value, far below the memory of the loaded model), so there is no load reading for the measured card during this speed test. What is recorded for the measured card: 2 MiB in use before the start, 15,626 MiB after the model had loaded, and 2 MiB after the stop. So I cannot show that nothing else used the box during the test.

python3 speed2.py --base-url http://127.0.0.1:38473/v1 --label byteshape-0911-iq4xs-speed-r3 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/byteshape-0911-iq4xs-speed-r3/byteshape-0911-iq4xs-speed-r3/speed2 --contract $RUN/byteshape-0911-iq4xs-speed-r3/byteshape-0911-iq4xs-speed-r3/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source byteshape/Qwen3.8-27B-GGUF README.md:120-127 @3fdfbd9b (thinking 1.0/0.95/20/0.0/presence 0.0; instruct (--reasoning off) 0.7/0.80/20/0.0/presence 1.5, repetition_penalty 1.0) and the models.json of the run tool (instruct-general, thinking-general); the same as the base model card Qwen/Qwen3.8-27B. My parts with thinking off use the instruct profile; every request carries its profile from this card --model qwen38-27b-byteshape-iq4xs --no-cache-prompt --server-log $HOME/byteshape-0911/logs/server.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source byteshape/Qwen3.8-27B-GGUF README.md:102 (-np 1 -fa on); the automatic slot count of the run tool's command does not load on 16 GB (my probe start, logs not published) --server-ctx 40960 --server-ctx-source the author's run tool byteshape.com/run-hf-model: context field, default 32768 (not a fixed value, minimum 512). I use 40960 (above the default): the largest tested window that fits with GPU-5 + MTP + f16 KV on this card (my probe starts: 40960 15626 MiB, 43008 15766 MiB, 45056 out of memory) and that allows LiveCodeBench (32768 output tokens) and RULER 32K. A 39504-token prompt was tested at 40960 (peak 15652 MiB)

How well it answers

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-09, except GPQA 30, thinking on: 2026-11-10.

Note from the run: MMLU-Pro (400 questions), GPQA Diamond (198), IFEval (200), LiveCodeBench (40 tasks) and RULER at 8K and 32K tokens (40 tasks each) ran with thinking off and the instruct sampling of the model card (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5), seed 42, one request at a time (the server has 1 slot). MMLU-Pro and GPQA Diamond ran with an answer limit of 48 tokens, letter only; IFEval with 4,096 tokens; LiveCodeBench with 32,768 tokens; RULER with 128 tokens. In the tests of the first run, 1 IFEval answer was cut off at its limit and counts as wrong; no other test of that run had a cut-off answer. LiveCodeBench answers are long even with thinking off: median 6,103 tokens, longest 30,841 (limit 32,768), and 2 of the 40 answers held no code. RULER by task: variable tracking 4 of 10 at 8K and 3 of 10 at 32K; multi-hop question answering 8 of 10 and 4 of 10. The thinking test (30 fixed GPQA Diamond questions, from the same public mirror) was added on 2026-11-10, as one separate run of this setup on a freshly started server: the records of that run show the same start command (40,960 tokens of context, 1 slot, MTP drafts), the same engine release and the same checksums of the test-tool files as the first run, and again one question at a time. It ran with thinking switched on by a request flag, the thinking sampling of the model card (temperature 1.0, top-p 0.95, top-k 20, min-p 0, presence penalty 0.0), seed 42 and an answer limit (cap) of 16,384 tokens. 7 of the 30 answers were cut off at the cap and count as wrong; the longest answer that finished had 15,173 tokens, the median answer 6,068 tokens, and the median time per question was 124.6 s (one question at a time). The answers file of the thinking test is published. These quality tests have no foreign-load guard; their sampling file lists the busiest other process every 10 seconds: 79 of 1,010 samples were above 25 % CPU (up to 654 %), 72 of them the indexing or mirror jobs of the box's own services, 4 the code sandbox of the LiveCodeBench test itself, 2 short find jobs, and 1 a script of mine that makes table images for posts (the peak of 654 %, in a single sample during the IFEval part). The memory of the measured card in use stayed between 15,658 and 15,674 MiB. GPQA Diamond: the original is gated and no Hugging Face token was available, so the public mirror fingertap/GPQA-Diamond (198 rows) was used in both runs. In the run of 2026-11-10 the same sampling file has 30 of 475 samples above 25 % CPU (up to 117 %), 28 of them the indexing or mirror jobs of the box's own services and 2 short jobs (find and git); the memory of the measured card in use stayed between 15,626 and 15,660 MiB. The answers file of LiveCodeBench is not published: the site's filter for power units flagged a letter count in the model's text (the number of times the letter W occurs in a puzzle string; it is not a power reading). The summary, the log and the contract of that test are published. These are my own tests, and no run of another setup is mixed into the numbers on this page; the thinking test was a separate run of this setup, as described above.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-byteshape-iq4xs-mtp-llamacpp-b12223-5060ti.json --parts mmlupro --job byteshape-0911-iq4xs

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-byteshape-iq4xs-mtp-llamacpp-b12223-5060ti.json --parts gpqa --job byteshape-0911-iq4xs

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-byteshape-iq4xs-mtp-llamacpp-b12223-5060ti.json --parts ifeval --job byteshape-0911-iq4xs

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-byteshape-iq4xs-mtp-llamacpp-b12223-5060ti.json --parts lcb --job byteshape-0911-iq4xs

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-byteshape-iq4xs-mtp-llamacpp-b12223-5060ti.json --parts ruler --job byteshape-0911-iq4xs

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-byteshape-iq4xs-mtp-llamacpp-b12223-5060ti.json --parts ruler --job byteshape-0911-iq4xs

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-byteshape-iq4xs-mtp-llamacpp-b12223-5060ti.json --parts think --job gpu1010-byteshape-think

How much context fits

Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed is measured on that answer at the largest size. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. First size that failed: 44K (45,056 tokens), it did not come up: not up after 900 s. Measured 2026-11-09. Raw files: result of every step · log · contract.

Note from the run: The server was restarted for every size with the context set to that size (one slot, f16 cache, MTP, no projector), and one prompt that fills about 95 % of the size was sent; the model had to answer. The sizes 4K, 8K, 16K, 32K, 40,960 and 43,008 loaded and answered. 45,056 did not: the server did not come up within 900 s (in my earlier probe start with this size it ran out of memory while reserving its compute buffers; the probe logs are not published). So the largest size that loaded and answered is 43,008 tokens, a little above the 40,960 used in the other tests. The 11.68 GiB in the row of 45,056 is the highest memory reading during that failed start. The memory number is the GPU memory in use on the whole card at the largest size (15.43 GiB; 12.85 GiB at 4K and 15.29 GiB at 40,960): the weights, the cache and the compute buffers. The decode speed shown (28.7 tok/s) is that of one short reply of 68 tokens after a prompt of 40,857 tokens. It depends on how many of the drafted tokens the model accepts (30 of 111 here, 40 of 75 in the 4K step, whose reply of 66 tokens ran at 46.0 tok/s), so it is not comparable with the decode speeds of the speed test. The prefill column is the server's own report for that one request.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-27b-byteshape-iq4xs-mtp-llamacpp-b12223-5060ti.json --parts fit --job byteshape-0911-iq4xs

How it plays the Gauntlet

One solo race, season seed 21, one attempt, no help.

Game rank as of Nov 9, 2026. The rank is the order of the Gauntlet finish times in this memory class. Race page and raw files.

The race ran the recipe named on its race page, with the sampling and answer limit of its lane config. The engine build of the race is not recorded in the race files.

Result (result.json) Race log (log.jsonl) Model answers (io.jsonl) System prompt Exact command Verify output Model config Race page

Also made this possible

• Engine (llama.cpp release b12223)ggml-org (llama.cpp)@ggml_org GitHub Hugging Face Website
• Base model (Qwen3.8-27B)QwenHugging Face @Alibaba_Qwen GitHub Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5060 Ti 16 GB (one GPU), the whole model in video memory. Intel Core i7-12700K, 32 GB RAM. Driver 595.71.05, kernel 6.12.86+deb13-amd64

Model

Qwen3.8-27B ByteShape ShapeLearn GPU-5 IQ4_XS-3.84bpw GGUF (byteshape/Qwen3.8-27B-GGUF @3fdfbd9b, 3.84 bpw, 13.08 GB)

Model, quant and recipe (GGUF files, revision 3fdfbd9b)

huggingface.co/byteshape/Qwen3.8-27B-GGUF

Base model (Qwen3.8-27B)

huggingface.co/Qwen/Qwen3.8-27B

Engine

llama.cpp b12223 (official release), one RTX 5060 Ti 16 GB, 40,960 tokens of context, 1 slot, text only (no vision projector) — the program that loads the model and answers requests · github.com/ggml-org/llama.cpp/releases/tag/b12223

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   CUDA_DEVICE_ORDER=PCI_BUS_ID \
     CUDA_VISIBLE_DEVICES=<GPU-UUID> llama-server \
     -m Qwen3.8-27B-IQ4_XS-3.84bpw.gguf \
     --no-mmproj \
     --alias qwen38-27b-byteshape-iq4xs \
     -ngl all \
     -c 40960 \
     -np 1 \
     -fa on \
     --spec-type draft-mtp \
     --spec-draft-n-max 3 \
     --reasoning off \
     --temp 0.7 \
     --top-p 0.8 \
     --top-k 20 \
     --min-p 0.0 \
     --presence-penalty 1.5 \
     --repeat-penalty 1.0 \
     --host 127.0.0.1 \
     --port 8888 \
     -fit off \
     --cache-ram 0 \
     --no-webui \
     --metrics

   What the parts mean

   • CUDA_DEVICE_ORDER=PCI_BUS_IDcounts the GPUs in slot order, the same order nvidia-smi shows
   • CUDA_VISIBLE_DEVICES=<GPU-UUID>which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints
   • -m Qwen3.8-27B-IQ4_XS-3.84bpw.ggufthe model file it loads
   • -c 40960the context window in tokens: how much text the model can see at once
   • -np 1how many requests it answers at the same time
   • -fa onturns on FlashAttention, a faster way to compute attention
   • --temp 0.7sampling temperature: how random the answers are
   • --top-p 0.8top-p sampling: only the most likely next tokens that together reach this probability are considered
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on
   • --cache-ram 0the most system RAM, in MiB, used to keep earlier prompts for reuse (0 turns this off)
   • -fit offturns off llama.cpp's automatic adjustment of unset settings to the free GPU memory; the command's settings are used as given
   • --no-webuiturns off the built-in chat web page; only the API answers
   • --metricsturns on a /metrics page with server statistics (Prometheus format)
   • --spec-…settings for speculative decoding: a draft guesses the next tokens and the model checks them (--spec-type: the kind of draft, --spec-draft-n-max: the most tokens drafted per step)
   • --alias qwen38-27b-byteshape-iq4xsthe model name clients use in their requests

llama.cpp b12223 (commit 0c6dca2fa; official release, Linux CUDA 13.5 build), MTP drafts (draft-mtp, up to 3 tokens), f16 cache, no vision projector; CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=<GPU-UUID> llama-server -m Qwen3.8-27B-IQ4_XS-3.84bpw.gguf --no-mmproj --alias qwen38-27b-byteshape-iq4xs -ngl all -c 40960 -np 1 -fa on --spec-type draft-mtp --spec-draft-n-max 3 --reasoning off --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --host 127.0.0.1 --port 8888 -fit off --cache-ram 0 --no-webui --metrics