Boole-32B-A2.5B-Preview Q2_0 on one RTX 5060 Ti 16 GB: Precision Numerics' llama.cpp fork

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-15. Raw files: speed result · speed contract · log · long answer.

What ran. Boole-32B-A2.5B-Preview (31.9B parameters, 2.5B active, a Mixture of Experts built on Flint by Recursive) in the Q2_0 file of Precision Numerics (8.96 GB; the model card puts the remaining experts at about 1.8 bits per weight), served by the llama.cpp fork of Precision Numerics at commit 7b225ec3; stock llama.cpp cannot load this model yet. The fork was built with the author's script, with CUDA 13.3 and architecture 120. The model always thinks: in the 512 tokens of the speed test there is only reasoning text, so the page has no words-per-second value and no '500 words' time. The 32,768-token window of the card's command is shared by prompt and reply, so the context steps at 32K, 64K and 128K and the prefill step at 32K do not fit and have no value; 16K fits. Where this run differs from the author's setup. (1) Build: this machine has no CUDA toolkit, so the author's build script ran in an NVIDIA PyTorch container (CUDA 13.3), and the server ran in the same container with only this card visible; the server command is the card's, unchanged. (2) The test server ran without --alias. (3) The author names no sampling for the server; the tests send the model card's evaluation values. The author publishes no number for a 16 GB card; the model card's table has the RTX 5090 only. We ran the author's llama-bench command here: decode 238.7 tok/s (238.8, 238.6, 238.7 in three separate processes) and prompt 5,025 tok/s (5,026, 5,025, 5,023). The speed test on this page measures the running server with streaming and reasoning text: 207.5 tok/s decode. The llama-bench runs are not part of the published raw files. The model with its 32,768-token window fits the card (9,830 MiB in use after loading, the card has 15,849 MiB); nothing was lowered.

python3 speed2.py --base-url http://127.0.0.1:44419/v1 --label boole-5060-1411-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/boole-5060-1411-speed-r0/boole-5060-1411-speed-r0/speed2 --contract $RUN/boole-5060-1411-speed-r0/boole-5060-1411-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --sampling {"temperature": 1.0, "top_p": 0.95, "top_k": 20} --sampling-source precision-numerics/Boole-32B-A2.5B-Preview-GGUF (HF @17e92b66, README.md) section 3 (evaluation settings: Temperature 1.0, top-p 0.95, top-k 20, 65,536-token context) and the base model Flint's recommendation (RecursiveSI/flint README: temperature 1.0, top-p 0.95, top-k 20). The author's server command sets no sampling; min_p and penalties are llama-server defaults. --model Boole-32B-A2.5B-Preview-Q2_0.gguf --no-cache-prompt --server-log <mnt>/backup/boole-1411/logs/server.log --engine-version-file $RUN/engine-version.txt --server-slots 4 --server-slots-source llama-server default (-np auto, --help: '-np N number of server slots (default: -1, -1 = auto)'; the author's command sets no -np). --server-ctx 32768 --server-ctx-source precision-numerics/Boole-32B-A2.5B-Preview-GGUF (HF @17e92b66, README.md) section 6 start command '-c 32768' and boole-demo README ('about 10.8 GB of GPU memory at this 32K context'). Prompt and reply share the window, so 32768-token prompts and the 64K/128K context depths cannot be sent (does_not_fit), the prefill depth 32K is left out of the prefill part for the same reason; nothing is raised by us.

How well it answers

Quality tests on this page ran with 1 request at a time, with the model's own always-on thinking, seed 42, the model card's sampling, and the fixed lists. Answer limits: 30,000 tokens for MMLU-Pro and GPQA, 4,096 for IFEval, 32,768 for LiveCodeBench, 128 for the long-text tasks, 16,384 for GPQA with thinking. A cut-off answer counts as wrong; the model thinks first, so the small limits of IFEval (4,096) and of the long-text tasks (128) leave it little room, and those numbers also measure the token limit. Long text 32K: the prompts have 32,770 tokens and do not fit the 32,768-token window of the card's command, so the 0.0 % is not a model result.

The model always thinks (the chat template has no switch). Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-15.

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5060ti.json --parts mmlupro --job boole-q1-5060-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5060ti.json --parts gpqa --job boole-q1-5060-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5060ti.json --parts ifeval --job boole-q1-5060-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5060ti.json --parts lcb --job boole-q1-5060-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5060ti.json --parts ruler --job boole-q1-5060-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5060ti.json --parts ruler --job boole-q1-5060-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5060ti.json --parts think --job boole-q1-5060-1411

Also made this possible

• Model files (Boole-32B-A2.5B-Preview, Q2_0 GGUF, revision 17e92b66)PPrecision Numerics@precision_num Hugging Face
• Base model (Flint)RRecursiveHugging Face
• llama.cpp, which the fork builds onGggml-org@ggml_org GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5060 Ti 16 GB (one machine, second card not used), driver 595.71.05, kernel 6.12.86

Model

Boole-32B-A2.5B-Preview (31.9B MoE, 2.5B active, base Flint) Q2_0 GGUF (precision-numerics/Boole-32B-A2.5B-Preview-GGUF @17e92b66, 8.96 GB, sha256 ea4c8f7f)

Model files (Boole-32B-A2.5B-Preview, Q2_0 GGUF, revision 17e92b66)

huggingface.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF

Base model (Flint)

huggingface.co/RecursiveSI/flint

Engine

llama.cpp fork of Precision Numerics at commit 7b225ec3, built with CUDA 13.3 — the program that loads the model and answers requests · github.com/PrecisionNumerics/llama.cpp/commit/7b225ec3c9acac279f0748c40136c5bf198acf9d

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   GGML_CUDA_GRAPH_OPT=1 llama-server \
     -m Boole-32B-A2.5B-Preview-Q2_0.gguf \
     -ngl 99 \
     -fa on \
     -ub 1024 \
     -c 32768 \
     --host 127.0.0.1 \
     --port 8888

   What the parts mean

   • -m Boole-32B-A2.5B-Preview-Q2_0.ggufthe model file it loads
   • -c 32768the context window in tokens: how much text the model can see at once
   • -ngl 99how many layers of the model go on the GPU (999 means all of them)
   • -fa onturns on FlashAttention, a faster way to compute attention
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on
   • -ub 1024the most tokens it sends to the GPU in one step (physical batch)

llama.cpp fork PrecisionNumerics/llama.cpp @7b225ec3 (author's pinned commit, build by scripts/setup_llama_cpp.sh, sm_120a), -fa on -ub 1024 -c 32768, GGML_CUDA_GRAPH_OPT=1, thinking always on; GGML_CUDA_GRAPH_OPT=1 llama-server -m Boole-32B-A2.5B-Preview-Q2_0.gguf -ngl 99 -fa on -ub 1024 -c 32768 --host 127.0.0.1 --port 8888 (the command of the model card, unchanged except host and port; fork Precision Numerics/llama.cpp at commit 7b225ec3, built with the author's script scripts/setup_llama_cpp.sh in boole-demo)