Diffuspace's Clover-35B-A3B-Preview on a 16 GB card: Q4_0 GGUF with llama.cpp, autoregressive mode

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K and 64K tokens and of 130,048 tokens for the 128K step, which is reduced so that prompt plus 512 output tokens fit the 131,072 window (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once (later supplementary run of 2026-11-05, same setup): C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-04 (requests at once: 2026-11-05). Raw files: speed result · speed contract · log · long answer. Requests at once, repeated on a freshly started server (the table of requests at once): speed result · speed contract · log.

Requests at once: the server has 4 slots (my setting; the README names no slot count). C8 exceeds the slots: the extra requests wait in the queue, and they were measured anyway. The speed run itself measured 1 request at once (total) at 26.9 tok/s; the supplementary run, which measured all levels in one session, measured 46.13 tok/s, and that value is shown. The speed run itself measured 2 at once (total) at 60.78 tok/s; the supplementary run, which measured all levels in one session, measured 59.86 tok/s, and that value is shown. The speed run itself measured 4 at once (total) at 69.8 tok/s; the supplementary run, which measured all levels in one session, measured 72.28 tok/s, and that value is shown. The speed run itself measured 8 at once (total) at 70.5 tok/s; the supplementary run, which measured all levels in one session, measured 71.76 tok/s, and that value is shown. The speed run itself measured 1 request at once (per request) at 26.9 tok/s; the supplementary run, which measured all levels in one session, measured 46.13 tok/s, and that value is shown. The speed run itself measured 2 at once (per request) at 30.41 tok/s; the supplementary run, which measured all levels in one session, measured 29.95 tok/s, and that value is shown. The speed run itself measured 4 at once (per request) at 17.47 tok/s; the supplementary run, which measured all levels in one session, measured 18.08 tok/s, and that value is shown. The speed run itself measured 8 at once (per request) at 11.95 tok/s; the supplementary run, which measured all levels in one session, measured 12.64 tok/s, and that value is shown. The speed test ran with the README of Diffuspace's GGUF repository (revision 32e22a02; Clover-35B-A3B-Preview-Q4_0.gguf, 18.24 GiB, sha256 checked against the repository's SHA256SUMS) and with the engine state that the README points to: llama.cpp pull request #31306 at commit cb09d4355 of 31 Oct 2026, built from source (CUDA 13.3.73 according to its build log, which is not published; sm_120a). I used llama-server instead of the README's example program, because the tests need an OpenAI-compatible endpoint. The README's recipe is self-speculative decoding (--spec-type draft-self). It is not used here, and this page runs the weights autoregressively, without drafts: with the experts in system RAM, llama-server in draft-self mode crashed on the first request (segmentation fault) in all seven starts I tried (the measurement setup at a context of 131,072 tokens and six variants of it at 32,768 tokens: 20 or 32 layers of experts in RAM, f16 or bf16 cache, flash attention on or off, batch 256 or 2048, draft length 2 or 15, 1 or 4 slots), and the README's own example program crashed the same way with 20 layers of experts in RAM; with 26 whole layers on the card it ran, at 6.8 to 7.9 tok/s in a probe (not a measurement). The maker's README lists the block-diffusion decoding of the model for its Hugging Face Transformers and SGLang implementations and lists llama.cpp as self-speculative only. So this is Clover's weights run as an ordinary autoregressive mixture-of-experts model. It is not the diffusion mode and not comparable 1:1 with setups of this model that use that mode or other weights. The merge commit of the pull request (e4b6c8ca6a, 2 Nov 2026, autoregressive only) cannot load this GGUF file ("key not found in model: clover_moe.expert_shared_count"), which is why the state before the rework is used; the README still says to fetch the pull request's head, which now is the reworked version. Other changes against the README's example: -c 131072 -np 4 -kvu (4 slots share one cache of 131,072 tokens; the README names no slot count and the server default is 1; the raw notes of the run call the 4 slots "from the author's recipe", which is not what the README says), -b 2048 -ub 2048 (the README names 256 and notes that its example program needs the prompt to fit within -b), --n-cpu-moe 20, no prompt cache. Sampling is the README's (temperature 0, top-k 1, seed 42); thinking is switched off by a request flag in every request (the prompt of the README's example ends with an opened thinking tag, so this differs from it); no cached tokens in the 61 requests that were checked. The headline 61.80 tok/s is the mean of the code prompt (61.68) and the prose prompt (61.92), medians of two runs each (spread 1.0 % and 0.1 %). Requests at once: the standard run gave 26.9 tok/s in total for one request at a time (all tokens divided by the wall time, including the prompt), far below the decode speed of about 62 tok/s. A probe afterwards (not a measurement) showed that the speed of the same short request depends on what the server did before: on a freshly started server the first request ran at 16.8 tok/s and the same request again at 62.6 tok/s, at 46.6 to 48.7 tok/s after prompts of 31,000 tokens, and at 60.8 tok/s after four requests at once. I think the unified cache of the server is the reason, but I have not proven it. So I repeated the requests-at-once test on a freshly started server, and the table shows that run (46.1, 59.9, 72.3 and 71.8 tok/s in total for 1, 2, 4 and 8 at once); the standard run gave 26.9, 60.8, 69.8 and 70.5. The decode speeds in the long-prompt table come from the standard run, on a server that had already handled many long prompts. These values vary between the two runs at 16K (spread 13.4 % for code, 15.5 % for prose; the speed contract flags the 16K prose value as noisy) and at 32K (9.3 % and 10.5 %). The memory test, on freshly started servers, shows higher decode speeds at 16K to 64K tokens (52.7, 48.3 and 39.5 tok/s; the replies had 62, 60 and 57 tokens, the limit was 96) and 28.0 tok/s at 128K (a reply of 50 tokens), about the same as the table. The 128K step of the long-prompt table ran with a prompt of 130,048 tokens: the prompt plus 512 output tokens must fit the window of 131,072. The load guard (a sample every half second) repeated 10 of the 38 steps (16 tries) because other processes used the CPU during the test: the box's own mirror jobs (rsync, git, find) and one copy of log files that I started myself. In 6 of the 38 kept steps the busiest other process still reached more than 25 % in single samples (git 166 %, 158 % and 144 %, a python3 process 282 %, find 44 %, a kernel thread 26 %); for a CPU load the guard throws a try away only when it lasts for two samples in a row, and the 128K step of the code prompt (run 1) kept its value after the maximum of 3 repeats. The wording clean try in the speed contract means only that this guard rule was met. The box also runs its own services, and the guard is set to ignore them (named in the speed contract, for example the search and indexing services); the busiest sample of the ten-second quiet check (not published) was the indexer of one of them at 140 % of one core. The GPU reading of the test tool pointed at the other GPU of this box (a known fault of the tool; I changed nothing), so there is no load reading for the measured card during this speed test; the repeated requests-at-once run read the same wrong GPU. What is recorded for the measured card: it had 2 MiB in use before the start and 15,038 MiB after the model had loaded (the same figure as in the memory test at this context), and 2 MiB after the stop. This box was therefore not quiet, and I cannot show that nothing else used the measured card during the test.

python3 speed2.py --base-url http://127.0.0.1:46111/v1 --label clover-box-5-0411-s-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/clover-box-5-0411-s-speed-r0/clover-box-5-0411-s-speed-r0/speed2 --contract $RUN/raw/clover-box-5-0411-s-speed-r0/clover-box-5-0411-s-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --sampling {"temperature": 0.0, "top_p": 1.0, "top_k": 1} --sampling-source Diffuspace/Clover-35B-A3B-Preview-GGUF README.md @32e22a02 (copy $HOME/bench/hf-gguf-README.md) :140 (--temp 0 --top-k 1 --seed 42, greedy). The checkpoint's generation_config.json (presence_penalty 0.9, repetition_penalty 1.05) is applied by the SGLang route (in our 1-Spark run of this model, which is not published) but NOT by llama.cpp (the GGUF README example sets no penalties), so none are sent here. --model clover-35b-a3b-preview --no-cache-prompt --server-log $HOME/clover-0411/logs/server.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 4 --server-slots-source author recipe names no slot count: the PR #31306 body server example (llama-server -m clover-bf16.gguf -ngl 99 -c 8192 -fa on) has no -np, and the llama-server default of this state is 1 slot (common/arg.cpp help text '-np: number of server slots (default: 1, -1 = auto)', cb09d4355). Test rule (largest setting the server allows at full context, named openly): we set -np 4 -kvu, 4 slots that share one unified KV pool of 131072 tokens (one request can use the whole pool; probe Nov 4, 2026 22:3x: 124,475-token prompt + 4 parallel requests ran with 15.2 of 16.3 GB VRAM). Same slot count as in our 1-Spark run of this model, which is not published (4, there --max-running-requests 4). --server-ctx 131072 --server-ctx-source GGUF metadata / README :53: supported context 131,072 tokens incl. prompt and response; server -c 131072 -np 4 -kvu (unified KV pool shared by the slots, so one request can use the whole 131072)

# Requests at once: supplementary run of 2026-11-05 (raw folder clover-box-5-0411-c-speed-r0-conc)
python3 speed2.py --base-url http://127.0.0.1:36639/v1 --label clover-box-5-0411-c-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/clover-box-5-0411-c-speed-r0/clover-box-5-0411-c-speed-r0/speed2 --contract $RUN/raw/clover-box-5-0411-c-speed-r0/clover-box-5-0411-c-speed-r0/speed2/speed-contract.json --parts conc --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --sampling {"temperature": 0.0, "top_p": 1.0, "top_k": 1} --sampling-source Diffuspace/Clover-35B-A3B-Preview-GGUF README.md @32e22a02 (copy $HOME/bench/hf-gguf-README.md) :140 (--temp 0 --top-k 1 --seed 42, greedy). The checkpoint's generation_config.json (presence_penalty 0.9, repetition_penalty 1.05) is applied by the SGLang route (in our 1-Spark run of this model, which is not published) but NOT by llama.cpp (the GGUF README example sets no penalties), so none are sent here. --model clover-35b-a3b-preview --no-cache-prompt --server-log $HOME/clover-0411/logs/server.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 4 --server-slots-source author recipe names no slot count: the PR #31306 body server example (llama-server -m clover-bf16.gguf -ngl 99 -c 8192 -fa on) has no -np, and the llama-server default of this state is 1 slot (common/arg.cpp help text '-np: number of server slots (default: 1, -1 = auto)', cb09d4355). Test rule (largest setting the server allows at full context, named openly): we set -np 4 -kvu, 4 slots that share one unified KV pool of 131072 tokens (one request can use the whole pool; probe Nov 4, 2026 22:3x: 124,475-token prompt + 4 parallel requests ran with 15.2 of 16.3 GB VRAM). Same slot count as in our 1-Spark run of this model, which is not published (4, there --max-running-requests 4). --server-ctx 131072 --server-ctx-source GGUF metadata / README :53: supported context 131,072 tokens incl. prompt and response; server -c 131072 -np 4 -kvu (unified KV pool shared by the slots, so one request can use the whole 131072) --supplement-of clover-box-5-0411-s-speed-r0

How well it answers

LiveCodeBench and the thinking test were not run on this setup, so the page shows no value for them.

Thinking off. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-04 to 2026-11-05.

Note from the run: MMLU-Pro (400 questions), GPQA Diamond (198), IFEval (200) and RULER at 8K and 32K tokens (40 tasks each) ran with greedy decoding (temperature 0, top-k 1, seed 42) and thinking off; 4 questions at a time, the slot count of the server. MMLU-Pro and GPQA Diamond ran with an answer limit of 1,024 tokens (my usual limit for these two is 48, so a model that adds a short reason after the letter is not counted as cut off); same questions, first-letter extraction. 5 answers were cut off at their limit and count as wrong: 1 of the 198 GPQA Diamond answers, 4 of the 200 IFEval answers (limit 4,096 tokens). RULER's variable-tracking task is the weak spot: 2 of 10 correct at 8K and 0 of 10 at 32K. LiveCodeBench and the thinking test were not run on this setup, so the page shows no value for them. These quality tests have no foreign-load guard; their sampling file (not published) lists the busiest other process every 10 seconds: 27 of 356 samples were above 25 % CPU (up to 197 %), 25 of them the indexer or re-index script of the box's own services, the rest short jobs such as a container listing and a git command. In this test the card itself was read: its memory in use stayed between 15,038 and 15,200 MiB, close to the 15,180 MiB peak of the memory test at this context. These are my own tests, and no run of another setup is mixed into the numbers on this page; the requests-at-once and memory tests were separate runs of this setup, as described above.

bash tools/quality2/measure_all.sh tools/quality2/cards/clover-35b-a3b-preview-q4_0-llamacpp-cb09d4355-ar-5060ti.json --parts mmlupro --job clover-box-5-0411-q

bash tools/quality2/measure_all.sh tools/quality2/cards/clover-35b-a3b-preview-q4_0-llamacpp-cb09d4355-ar-5060ti.json --parts gpqa --job clover-box-5-0411-q

bash tools/quality2/measure_all.sh tools/quality2/cards/clover-35b-a3b-preview-q4_0-llamacpp-cb09d4355-ar-5060ti.json --parts ifeval --job clover-box-5-0411-q

bash tools/quality2/measure_all.sh tools/quality2/cards/clover-35b-a3b-preview-q4_0-llamacpp-cb09d4355-ar-5060ti.json --parts ruler --job clover-box-5-0411-q

bash tools/quality2/measure_all.sh tools/quality2/cards/clover-35b-a3b-preview-q4_0-llamacpp-cb09d4355-ar-5060ti.json --parts ruler --job clover-box-5-0411-q

How much context fits

Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed is measured on that answer at the largest size. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. Measured 2026-11-05. Raw files: result of every step · log · contract.

Note from the run: The server was restarted for every size with the context set to that size (the experts of the first 20 layers stay in system RAM), and one prompt that fills about 95 % of the size was sent; the model had to answer. All six sizes from 4K to 128K loaded and answered, so the largest size tested, 131,072 tokens, is the highest size in my list and the context of the other tests. The memory number is the GPU memory in use on the whole card at the largest size (14.82 GiB; 11.91 GiB at 4K, 13.32 GiB at 64K): the weights that are not in system RAM, the cache and the compute buffers. The prefill column is what the server reported; it is not a key figure and not comparable with the speed test, because the first request after a server start probably also reads the weights from disk. The decode speed shown (28.0 tok/s) is that of a reply of 50 tokens at the largest size (the limit was 96 tokens); the same value for the 4K size, where the reply used all 96 tokens, is 59.24 tok/s.

bash tools/quality2/measure_all.sh tools/quality2/cards/clover-35b-a3b-preview-q4_0-llamacpp-cb09d4355-ar-5060ti.json --parts fit --job clover-box-5-0411-f

Also made this possible

• Engine (llama.cpp pull request #31306, state cb09d4355)ggml-org (llama.cpp)@ggml_org GitHub Hugging Face Website
• Base model (Qwen3.6-35B-A3B)QwenHugging Face @Alibaba_Qwen GitHub Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5060 Ti 16 GB (one GPU) with the MoE experts of the first 20 layers in system RAM (--n-cpu-moe 20). Intel Core i7-12700K, 32 GB RAM. Driver 595.71.05, kernel 6.12.86+deb13-amd64

Model

Diffuspace/Clover-35B-A3B-Preview-GGUF @32e22a02 (diffusion LLM on Qwen3.6-35B-A3B, 34.7B total / ~3B active, block size 32), file Clover-35B-A3B-Preview-Q4_0.gguf (sha256 3e2cf14c, 19.59 GB)

Model and recipe (GGUF files, revision 32e22a02)

huggingface.co/Diffuspace/Clover-35B-A3B-Preview-GGUF

Engine

llama.cpp pull request #31306 at commit cb09d4355, one RTX 5060 Ti 16 GB with the experts of the first 20 layers in system RAM, 131,072 tokens of context, 4 slots — the program that loads the model and answers requests · github.com/ggml-org/llama.cpp/pull/31306

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   docker run \
     --gpus device=1 \
     --memory 20g ... llama-server \
     -m Clover-35B-A3B-Preview-Q4_0.gguf \
     -a clover-35b-a3b-preview \
     --host 127.0.0.1 \
     --port 8888 \
     --jinja \
     -ngl 99 \
     --n-cpu-moe 20 \
     -fit off \
     -fa on \
     -ctk bf16 \
     -ctv bf16 \
     -c 131072 \
     -np 4 \
     -kvu \
     -b 2048 \
     -ub 2048 \
     -t 8 \
     -tb 8 \
     --temp 0 \
     --top-k 1 \
     --seed 42 \
     --no-webui \
     --metrics \
     --cache-ram 0

   What the parts mean

   • -m Clover-35B-A3B-Preview-Q4_0.ggufthe model file it loads
   • -c 131072the context window in tokens: how much text the model can see at once
   • -ngl 99how many layers of the model go on the GPU (a number above the layer count means all of them)
   • --n-cpu-moe 20keeps the expert weights of this many layers in system RAM, so a model bigger than the GPU memory still runs
   • -np 4how many requests it answers at the same time
   • --jinjauses the chat template that comes with the model
   • -fa onturns on FlashAttention, a faster way to compute attention
   • --temp 0sampling temperature: how random the answers are
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on
   • -kvuone shared KV cache (context memory) for all parallel requests, instead of a fixed part per request
   • --cache-ram 0the most system RAM, in MiB, used to keep earlier prompts for reuse (0 turns this off)
   • -fit offturns off llama.cpp's automatic adjustment of unset settings to the free GPU memory; the command's settings are used as given
   • -t 8how many CPU threads it uses
   • -b 2048batch size: the most prompt tokens it processes in one go (logical batch)
   • -ub 2048the most tokens it sends to the GPU in one step (physical batch)
   • --no-webuiturns off the built-in chat web page; only the API answers
   • --metricsturns on a /metrics page with server statistics (Prometheus format)

llama.cpp PR #31306 pre-merge state cb09d4355 (Oct 31, 2026, 5 commits on f4d8af608a: Clover model + draft-self + server fix; a state that loads the official GGUF; the later merge commit does not) (source build, CUDA 13.3 container, sm_120a), autoregressive decoding, no --spec-type, bf16 KV; docker run --gpus device=1 --memory 20g ... llama-server -m Clover-35B-A3B-Preview-Q4_0.gguf -a clover-35b-a3b-preview --host 127.0.0.1 --port 8888 --jinja -ngl 99 --n-cpu-moe 20 -fit off -fa on -ctk bf16 -ctv bf16 -c 131072 -np 4 -kvu -b 2048 -ub 2048 -t 8 -tb 8 --temp 0 --top-k 1 --seed 42 --no-webui --metrics --cache-ram 0