Empero's Qwen3.8-35B-A3B-Distill on a 16 GB card: Q4_K_M GGUF with llama.cpp, the expert weights of 27 layers in system RAM

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds; long prompts (later supplementary run of 2026-11-11, same setup): prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests. Measured 2026-11-10 (long prompts: 2026-11-11). Raw files: speed result · speed contract · log · long answer · speed result · speed contract · log.

Requests at once: the server has 4 slots (my choice, the maker names none). C8 exceeds the slots: the extra requests wait in the queue, and they were measured anyway. Empero (the maker) gives no server recipe. Its GGUF model card (revision b1f9d1dc; line numbers are those of its README.md) lists Q4_K_M as the file marked "Recommended" (line 44), asks for "a recent llama.cpp build with Qwen3.6 / Gated DeltaNet MoE support" (line 33) and shows one usage line, llama-cli -m Qwen3.8-35B-A3B-Q4_K_M.gguf --temp 0.6 --top-p 0.95 --top-k 20 -n 16384 -cnv (lines 75 to 77), with the remark that the model is a reasoning model whose every answer opens with a <think> block (line 80). The card names no context size, no cache type, no server flags and no slot count, and it gives no speed number. The file is Qwen3.8-35B-A3B-Q4_K_M.gguf, 21,713,462,944 bytes (20.22 GiB), sha256 196103269085bc54c9b8f49ed21e9f53e1b56b465e8b796c6d8e31e06f63cfa5, which is the value that Hugging Face lists for the file at that revision. My server command: llama-server with all layers offloaded (-ngl all) but the expert weights of the first 27 layers kept in system RAM (--n-cpu-moe 27), -lm none, -fit off, flash attention, a window of 262,144 tokens with 4 slots that share one unified cache (-c 262144 -np 4 -kvu), batch sizes 2048 (-b 2048 -ub 2048), 8 threads (-t 8 -tb 8), --jinja, the maker's sampling (--temp 0.6 --top-p 0.95 --top-k 20), --seed 42, --cache-ram 0, --no-webui and --metrics, host and port. The deviations from the maker's usage line are all mine and are stated here: the offload of the experts is forced, because the model file (20.22 GiB) does not fit into the 16 GB of the card (in a probe before the measurements, --n-cpu-moe 24 did not load, with a compute buffer of 1540 MiB out of memory, and 27 did); -lm none, which I set because the model file lies on a hard disk; thinking is off in the speed test (the maker's mode is thinking); no min-p (the maker names none); f16 cache; the window of 262,144 and the 4 slots with a unified cache are my choice (the maker names no server line; the llama-server default would be 1 slot). The engine is the official llama.cpp release b12223 (commit 0c6dca2fa, package llama-b12223-bin-ubuntu-cuda-13.5-x64, its sha256 equal to the GitHub digest), not a build of my own. No vision projector is loaded, so the page measures text only. Not measured: the other quants of the repository, the vision input, an MTP or draft setup, and a quantised cache, so this page makes no comparison between quants. The headline 63.4 tok/s is the mean of the code prompt (63.8) and the prose prompt (63.0), medians of two runs each (spread 1.4 % and 0.0 %, as in the table). There is no draft acceptance rate: the setup has no speculative decoding. The speed test sent the maker's sampling with thinking off in every request: the answer of the one long-answer test (4,879 tokens) ended by itself, and the tool's cache check found no cached prompt tokens in any of the 61 requests it examined. Requests at once: the server has 4 slots (my choice), which share one cache, so 1, 2 and 4 requests run in their own slots and the 8 requests queue for them (the tool measured the level of 8 anyway). The total is 46.1, 56.5, 65.6 and 65.0 tok/s for 1, 2, 4 and 8 requests at once; the speed of one request falls from 46.1 to 28.3, 16.4 and 11.0 tok/s. The 1-request figure includes the prompt time, so it is below the decode speed. These are measured rates for this setup, not a statement about how the rate would scale on other hardware. Long prompts: all four steps (16K, 32K, 64K and 128K tokens) ran at the window of 262,144 tokens, and the long-prompt table shows the repeat of this part, a run of its own on 11 Nov 2026 (14:21 to 14:43) with the same start command, the same engine, the same weights and the same sampling file as the speed run (I compared the records: they are identical). All four steps have two valid runs of the code prompt and two of the prose prompt, so every cell is filled. The 128K step is a full 131,072 tokens: its first token came after 164.7 s (prefill 795.7 tok/s), and decode at that depth is 44.2 tok/s (code) and 44.4 tok/s (prose), against 58.4 and 57.8 tok/s at 32K. The first long-prompt run, inside the speed run of 10 Nov, had no valid code decode value at 16K: its first code run ended after 23 tokens (finish reason "stop"), which is below the tool's minimum of 256 tokens for a valid decode run, and the tool listed that step as failed (field "failed" in the summary of that run). That is why the part was repeated. The raw files of the first long-prompt run stay published in the folder of the speed run, and the page shows no number from it. The headline prefill figures are for 2K, 8K and 32K prompts. The maker's own numbers (its main model card, lines 51 to 63) are MMLU, ARC-Challenge and ARC-Easy by log-likelihood scoring with lm-evaluation-harness, which my tests do not repeat, so my numbers neither confirm nor contradict them. The card says the student was trained on examples of 8,192 tokens (main card, lines 96 to 97: "so behaviour on long-form generation and long-context workloads may be degraded relative to the base"); the long-prompt steps and the two long-text tests on this page are measurements at depths far above that. The first speed run of this setup (9 Nov 2026, 22:47 to 22:52) was stopped by the load guard of the test tool and counted as failed: during the prefill of the 32K prompt the kernel's memory-compaction thread (kcompactd0) used more than 25 % of a core in two samples in a row, up to 100 %, and the step was still disturbed in the last of its 3 repeats; the code decode of that run also had a run of only 23 tokens. Its raw files are not published, because that run has no register row. After that I added kcompactd to the guard's list of processes it ignores. This is my own reading, not a measurement: the thread's work comes from the setup itself (the expert weights kept in system RAM on a machine with 32 GB of RAM that also runs services of its own), not from a program of someone else. The speed run shown is the repeat (10 Nov, 02:09 to 02:40). Its guard (a sample every 0.5 s, step repeated after 2 samples in a row above 25 % of a core, the box's own background services and kcompactd on its ignore list) repeated none of its 38 steps in 3,717 samples; the busiest other process in any single step was a git backup job of the box's own files at 150.0 % of a core, in the second code run of the 128K step. The tool's separate quiet check (one sample every 10 seconds, no ignore list) saw 8 of 185 samples above 25 % of one core during the run (up to 760.0 %; 5 of them the box's own indexing service, 1 an index rebuild of the box's own services, 1 a git backup job and 1 a short Python helper). Nothing was repeated or removed. The GPU reading of that quiet check is not that of the measured card: it stays at one constant value of 5,602 MiB, which is the memory use of another GPU in the machine, so there is no memory reading of the measured card during the speed test. The repeat of the long-prompt part (11 Nov) ran with the same guard setting: it repeated none of its 16 steps in 2,700 samples, and the busiest other process in any single step was a git backup job of the box's own files at 132.0 % of a core, in the second code run of the 32K step. The quiet check of that run (one sample every 10 seconds, no ignore list) saw 7 of 135 samples above 25 % of one core (up to 111.0 %): 4 in the 64K step (an index rebuild of the box's own services at 97.2 %, and the box's own indexing service at 67.5, 56.5 and 60.1 %) and 3 in the 128K step (the box's own indexing service at 111.0, 101.0 and 61.1 %). Nothing was repeated or removed. Its GPU reading is again the constant value of 5,602 MiB of another GPU, so there is no memory reading of the measured card for this run either.

python3 speed2.py --base-url http://127.0.0.1:40967/v1 --label qwen38-35b-distill-0911-speed-r2 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/qwen38-35b-distill-0911-speed-r2/qwen38-35b-distill-0911-speed-r2/speed2 --contract $RUN/qwen38-35b-distill-0911-speed-r2/qwen38-35b-distill-0911-speed-r2/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --sampling {"temperature": 0.6, "top_p": 0.95, "top_k": 20} --sampling-source empero-ai/Qwen3.8-35B-A3B-Distill-GGUF README.md @b1f9d1dc lines 75-77 (llama-cli --temp 0.6 --top-p 0.95 --top-k 20) and line 96 (temperature=0.6, top_p=0.95, top_k=20); main model card empero-ai/Qwen3.8-35B-A3B-Distill line 90 (the same values; greedy decoding is a known repetition-loop failure mode). The maker has one profile only (thinking mode), so the instruct and the thinking profile are the same here. No min-p and no penalties (the maker names none). My parts with thinking off send enable_thinking=false (the chat template of the GGUF file), which is not the maker's mode --model qwen38-35b-a3b-distill-q4km --no-cache-prompt --server-log $HOME/qwen38-35b-0911/logs/server.log --engine-version-file $RUN/engine-version.txt --server-slots 4 --server-slots-source my choice: the maker names no server slots (its README lines 75-77 show only llama-cli); the llama-server default would be 1 slot. I use the largest setting at the full window: -np 4 -kvu, four slots that share one unified cache of 262144 tokens --server-ctx 262144 --server-ctx-source GGUF metadata qwen35moe.context_length 262144; main model card line 49 'Context Length: 262,144 natively'. Server -c 262144 -np 4 -kvu (one unified cache: one request may use all of it), f16 cache (the llama.cpp default; the maker names no cache type). Probe on 9 Nov 2026, 22:37 to 22:44 (not a measurement): --n-cpu-moe 24 does not load (compute buffer of 1540 MiB out of memory), --n-cpu-moe 27 loads (video memory peak 14688 MiB, decode 59.5 tok/s, 16K prefill 1306 tok/s); for comparison -c 131072 with --n-cpu-moe 22: 13902 MiB, 70.9 tok/s, 1613 tok/s. I use the full native window

# Long prompts: supplementary run of 2026-11-11 (raw folder distill-ctx-1111-speed-r0-ctx)
python3 speed2.py --base-url http://127.0.0.1:32859/v1 --label distill-ctx-1111-speed-r0 --prompts $BENCH/repos/<speed-test-tool>/prompts-v2.json --output $RUN/distill-ctx-1111-speed-r0/distill-ctx-1111-speed-r0/speed2 --contract $RUN/distill-ctx-1111-speed-r0/distill-ctx-1111-speed-r0/speed2/speed-contract.json --parts ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --sampling {"temperature": 0.6, "top_p": 0.95, "top_k": 20} --sampling-source empero-ai/Qwen3.8-35B-A3B-Distill-GGUF README.md @b1f9d1dc lines 75-77 (llama-cli --temp 0.6 --top-p 0.95 --top-k 20) and line 96 (temperature=0.6, top_p=0.95, top_k=20); main model card empero-ai/Qwen3.8-35B-A3B-Distill line 90 (the same values; greedy decoding is a known repetition-loop failure mode). The maker has one profile only (thinking mode), so the instruct and the thinking profile are the same here. No min-p and no penalties (the maker names none). My parts with thinking off send enable_thinking=false (the chat template of the GGUF file), which is not the maker's mode --model qwen38-35b-a3b-distill-q4km --no-cache-prompt --server-log $HOME/qwen38-35b-0911/logs/server.log --engine-version-file $RUN/engine-version.txt --server-slots 4 --server-slots-source my choice: the maker names no server slots (its README lines 75-77 show only llama-cli); the llama-server default would be 1 slot. I use the largest setting at the full window: -np 4 -kvu, four slots that share one unified cache of 262144 tokens --server-ctx 262144 --server-ctx-source GGUF metadata qwen35moe.context_length 262144; main model card line 49 'Context Length: 262,144 natively'. Server -c 262144 -np 4 -kvu (one unified cache: one request may use all of it), f16 cache (the llama.cpp default; the maker names no cache type). Probe on 9 Nov 2026, 22:37 to 22:44 (not a measurement): --n-cpu-moe 24 does not load (compute buffer of 1540 MiB out of memory), --n-cpu-moe 27 loads (video memory peak 14688 MiB, decode 59.5 tok/s, 16K prefill 1306 tok/s); for comparison -c 131072 with --n-cpu-moe 22: 13902 MiB, 70.9 tok/s, 1613 tok/s. I use the full native window --supplement-of qwen38-35b-distill-0911

How well it answers

All seven quality tests ran. Six ran with thinking off and one (GPQA 30) with thinking on; the maker's mode is thinking, so the six may understate the model. Answers cut off at the token limit count as wrong: 5 of 200 in IFEval, 8 of 40 in LiveCodeBench and 3 of 30 in GPQA 30. The notes below say how the numbers were taken.

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-09 to 2026-11-10.

Note from the run: MMLU-Pro (400 questions), GPQA Diamond (198), IFEval (200), LiveCodeBench (40 tasks) and RULER at 8K and 32K tokens (40 tasks each) ran with thinking off and the maker's sampling (temperature 0.6, top-p 0.95, top-k 20), seed 42, 4 questions in flight at a time (the server has 4 slots). The answer limits are 1,024 tokens for MMLU-Pro and GPQA Diamond (the tool's default is 48; I raised it so that a model that writes an explanation after the letter is not counted as cut off; the answer is still read as the first letter, the median answer was 2 tokens in both tests, and none was cut off), 4,096 for IFEval, 32,768 for LiveCodeBench and 128 for RULER. GPQA 30 ran with thinking on, an answer limit of 16,384 tokens and 4 questions at a time. GPQA Diamond ran from the public mirror fingertap/GPQA-Diamond (198 rows), because the original is gated and no Hugging Face token was available.

Thinking: the maker's model is a reasoning model, every answer opens with a thinking block, and the maker's card has one sampling profile for it (thinking mode). Six of my seven tests run with thinking off, so they are not the setting the maker describes and may understate the model; I have not measured by how much. Only GPQA 30 ran with thinking on: 22 of 30. The median answer of that test used 1,578 tokens and 68.3 seconds per question (measured with 4 questions at a time), and 3 of the 30 answers were cut off at the limit of 16,384 tokens.

Cut-off answers count as wrong: 0 in MMLU-Pro, 0 in GPQA Diamond, 5 of 200 in IFEval (limit 4,096 tokens), 8 of the 40 in LiveCodeBench (limit 32,768 tokens; the median answer used only 1,202 tokens, so the cut-off answers are long outliers) and 0 in long text (the RULER tests). With thinking, 3 of the 30 GPQA answers were cut off. The register's cut-off total of 8 counts only the parts of its own run (here LiveCodeBench); the 5 cut-off IFEval answers are in the summary of the first run. RULER by task: multi-key retrieval 10 of 10 at 8K and 10 of 10 at 32K; common-word counting 9 of 10 and 10 of 10; multi-hop question answering 9 of 10 and 3 of 10; variable tracking 0 of 10 at both lengths.

The quality numbers come from four runs of this setup, all with the same server command. The first quality run (9 Nov 2026, 22:52) was stopped by hand at 23:56, during LiveCodeBench, because the machine was needed for another test. Its published contract records that LiveCodeBench failed (broken pipe) and that RULER could not connect (connection refused). MMLU-Pro, GPQA Diamond and IFEval had finished completely before that (400, 198 and 200 answers), and the table shows their results; the tool wrote a second contract of that run, marked as recovered, which keeps only these three complete parts (that file is not published). RULER ran again on 10 Nov at 02:40 and LiveCodeBench at 05:21; the memory test ran in between. GPQA 30 with thinking ran last, as a run of its own (10 Nov, 12:58 to 13:15) with the same start command, the same engine, the same weights and the same sampling file as the other runs (I compared the records of both: they are identical). The table mixes these runs part by part and mixes no other run in.

The LiveCodeBench answers file is not published: the site's filter for power readings held it back (a program in one answer has a variable named power; it is not a power reading). The summary, the log and the contract of that test are published, and the numbers of the table were counted from the complete file. That file has 44 lines: 40 final records, one per task, and 4 error records (2 connection resets and 2 connections closed without an answer) of requests that were asked again; each of those tasks has a later complete record, and the summary counts the final record of each task (0 errors).

Foreign load: the quality tests have no guard; their sampling file lists the busiest other process every 10 seconds. First run (MMLU-Pro, GPQA, IFEval and the interrupted start of LiveCodeBench): 21 of 380 samples above 25 % of one core (up to 194.0 %; 16 the box's own indexing service, 2 a git backup job of the box's own files, 2 an index rebuild of the box's own services and 1 a short Python helper). RULER: 12 of 188 (up to 477.0 %; 9 the indexing service, 2 a git backup job, 1 an index rebuild). LiveCodeBench: 83 of 296 (up to 410.0 %; 48 video download and conversion jobs of the box's own tools, 30 the indexing service, 2 file searches, 1 a git backup job, 1 a container runtime process, and 1 the program of the code test itself). GPQA 30 with thinking: 4 of 101 (up to 106.0 %; 3 the indexing service, 1 an index rebuild). The memory in use on the measured card stayed between 14,548 and 14,698 MiB in all four runs. None of this was repeated or removed.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-35b-a3b-distill-q4km-llamacpp-b12223-5060ti.json --parts mmlupro --job qwen38-35b-distill-0911

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-35b-a3b-distill-q4km-llamacpp-b12223-5060ti.json --parts gpqa --job qwen38-35b-distill-0911

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-35b-a3b-distill-q4km-llamacpp-b12223-5060ti.json --parts ifeval --job qwen38-35b-distill-0911

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-35b-a3b-distill-q4km-llamacpp-b12223-5060ti.json --parts lcb --job qwen38-35b-distill-0911

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-35b-a3b-distill-q4km-llamacpp-b12223-5060ti.json --parts ruler --job qwen38-35b-distill-0911

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-35b-a3b-distill-q4km-llamacpp-b12223-5060ti.json --parts ruler --job qwen38-35b-distill-0911

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-35b-a3b-distill-q4km-llamacpp-b12223-5060ti.json --parts think --job gpu1010-distill-think

How much context fits

Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed is measured on that answer at the largest size. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. First size that failed: 256K (262,144 tokens), RemoteDisconnected: Remote end closed connection without response. Measured 2026-11-10. Raw files: result of every step · log · contract.

Note from the run: The server was restarted for every size with the context set to that size (4 slots, unified cache, f16 cache), and one prompt that fills about 95 % of the size was sent (prompts of 3,891 to 124,518 tokens up to the largest step that answered, 96 answer tokens at most); the model had to answer. The sizes 4K, 8K, 16K, 32K, 64K and 131,072 loaded and answered, so 131,072 tokens is the largest size that answered. The step of 262,144 tokens, the native window and the window of my speed and quality tests, loaded (the server was up after 118.5 s, with 14.34 GiB of GPU memory at its peak), but the request with the long prompt got no answer: the connection ended after 41.2 seconds with "Remote end closed connection without response", and the test stopped there. I do not know why: the server's own output was not kept (the log of that step holds only a start line and a stop line of the test script), so it is not recorded whether the server stopped or what it ran out of. This means that a prompt of about 95 % of 262,144 tokens is not shown to work on this setup; the speed test did answer prompts of 131,072 tokens at the 262,144 window (in both of its long-prompt runs, 10 and 11 Nov), and nothing between 131,072 and 262,144 was tried. The memory number is the GPU memory in use on the whole card (the peak of the test): 8.43 GiB at 4K and 11.34 GiB at 131,072 tokens; the expert weights in system RAM are not part of it. The decode speed shown (41.0 tok/s at 131,072) is that of one short reply of 77 tokens after a prompt of 124,518 tokens; it is not comparable with the decode speeds of the speed test, and the replies of the sizes differ in length (96 tokens at 4K, ended by the limit of 96). The prefill column is the server's own report for that one request.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwen38-35b-a3b-distill-q4km-llamacpp-b12223-5060ti.json --parts fit --job qwen38-35b-distill-0911

Also made this possible

• Engine (llama.cpp release b12223, the official Linux CUDA 13.5 package)ggml-org (llama.cpp)@ggml_org GitHub Hugging Face Website
• Base model (Qwen3.6-35B-A3B, the architecture the distillation was trained into)QwenHugging Face @Alibaba_Qwen GitHub Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5060 Ti 16 GB (one GPU). The model file (20.22 GiB) is larger than the video memory, so the expert weights of the first 27 layers stay in system RAM (--n-cpu-moe 27) and the rest of the model is in video memory. Intel Core i7-12700K, 32 GB RAM. Driver 595.71.05, kernel 6.12.86+deb13-amd64

Model

Qwen3.8-35B-A3B-Distill (Empero, empero-ai/Qwen3.8-35B-A3B-Distill-GGUF @b1f9d1dc, Distillation in Qwen3.6-35B-A3B, 35B total / ~3B active, 21.71 GB)

Model, distillation and GGUF quants (file Qwen3.8-35B-A3B-Q4_K_M.gguf, revision b1f9d1dc; the author of both repositories is empero-ai)

huggingface.co/empero-ai/Qwen3.8-35B-A3B-Distill-GGUF/tree/b1f9d1dcc3de8aa867669b0ab919384aeeb9b8d5

Base model (Qwen3.6-35B-A3B, the architecture the distillation was trained into)

huggingface.co/Qwen/Qwen3.6-35B-A3B

Engine

llama.cpp b12223 (official release), one RTX 5060 Ti 16 GB, 262,144 tokens of context, 4 slots, text only (no vision projector) — the program that loads the model and answers requests · github.com/ggml-org/llama.cpp/releases/tag/b12223

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   CUDA_DEVICE_ORDER=PCI_BUS_ID \
     CUDA_VISIBLE_DEVICES=<GPU1-UUID> llama-server \
     -m Qwen3.8-35B-A3B-Q4_K_M.gguf \
     --alias qwen38-35b-a3b-distill-q4km \
     -ngl all \
     --n-cpu-moe 27 \
     -lm none \
     -fit off \
     -fa on \
     -c 262144 \
     -np 4 \
     -kvu \
     -b 2048 \
     -ub 2048 \
     -t 8 \
     -tb 8 \
     --jinja \
     --temp 0.6 \
     --top-p 0.95 \
     --top-k 20 \
     --seed 42 \
     --host 127.0.0.1 \
     --port 8888 \
     --cache-ram 0 \
     --no-webui \
     --metrics

   What the parts mean

   • CUDA_DEVICE_ORDER=PCI_BUS_IDcounts the GPUs in slot order, the same order nvidia-smi shows
   • CUDA_VISIBLE_DEVICES=<GPU1-UUID>which NVIDIA GPU the program may use: the ID of the GPU, its number or the UUID that nvidia-smi -L prints
   • -m Qwen3.8-35B-A3B-Q4_K_M.ggufthe model file it loads
   • -c 262144the context window in tokens: how much text the model can see at once
   • --n-cpu-moe 27keeps the expert weights of this many layers in system RAM, so a model bigger than the GPU memory still runs
   • -np 4how many requests it answers at the same time
   • --jinjauses the chat template that comes with the model
   • -fa onturns on FlashAttention, a faster way to compute attention
   • --temp 0.6sampling temperature: how random the answers are
   • --top-p 0.95top-p sampling: only the most likely next tokens that together reach this probability are considered
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on
   • -kvuone shared KV cache (context memory) for all parallel requests, instead of a fixed part per request
   • --cache-ram 0the most system RAM, in MiB, used to keep earlier prompts for reuse (0 turns this off)
   • -fit offturns off llama.cpp's automatic adjustment of unset settings to the free GPU memory; the command's settings are used as given
   • -t 8how many CPU threads it uses
   • -b 2048batch size: the most prompt tokens it processes in one go (logical batch)
   • -ub 2048the most tokens it sends to the GPU in one step (physical batch)
   • --no-webuiturns off the built-in chat web page; only the API answers
   • --metricsturns on a /metrics page with server statistics (Prometheus format)
   • --alias qwen38-35b-a3b-distill-q4kmthe model name clients use in their requests

llama.cpp b12223 @0c6dca2fa (official release ubuntu-cuda-13.5-x64), autoregressive (no MTP, no draft model), f16 KV, --n-cpu-moe 27, -lm none; CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=<GPU1-UUID> llama-server -m Qwen3.8-35B-A3B-Q4_K_M.gguf --alias qwen38-35b-a3b-distill-q4km -ngl all --n-cpu-moe 27 -lm none -fit off -fa on -c 262144 -np 4 -kvu -b 2048 -ub 2048 -t 8 -tb 8 --jinja --temp 0.6 --top-p 0.95 --top-k 20 --seed 42 --host 127.0.0.1 --port 8888 --cache-ram 0 --no-webui --metrics