Boole-32B-A2.5B-Preview Q2_0 on one RTX 5090: Precision Numerics' llama.cpp fork

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-15. Raw files: speed result · speed contract · log · long answer.

What ran. Boole-32B-A2.5B-Preview (31.9B parameters, 2.5B active, a Mixture of Experts built on Flint by Recursive) in the Q2_0 file of Precision Numerics (8.96 GB; the model card puts the remaining experts at about 1.8 bits per weight), served by the llama.cpp fork of Precision Numerics at commit 7b225ec3; stock llama.cpp cannot load this model yet. The fork was built with the author's script, with CUDA 13.3 and architecture 120. The model always thinks: in the 512 tokens of the speed test there is only reasoning text, so the page has no words-per-second value and no '500 words' time. The 32,768-token window of the card's command is shared by prompt and reply, so the context steps at 32K, 64K and 128K and the prefill step at 32K do not fit and have no value; 16K fits. Where this run differs from the author's setup. (1) Build: CUDA 13.3 instead of 12.8, because the default toolkit on this machine (12.8) fails to compile with glibc 2.41; the author names 12.8 or newer. (2) llama-bench was built in addition (the author's script does not build it). (3) The test server ran without --alias, so the model name in the requests is the file name. (4) The author names no sampling for the server; the tests send the model card's evaluation values (temperature 1.0, top-p 0.95, top-k 20). Against the author's own number. The model card gives 565.8 tok/s decode (±0.4) and 18,297 tok/s prompt (±42) for the RTX 5090 with llama-bench; we ran the same llama-bench command (GGML_CUDA_GRAPH_OPT=1, -ngl 99 -fa on -ub 1024, 512 prompt and 128 output tokens, 5 repeats, three separate processes): decode 567.0 tok/s (±0.63, ±0.73, ±0.67 in the three processes: 567.0, 567.0, 566.9) and prompt 18,065, 18,101 and 18,122 tok/s. The speed test on this page measures the running server with streaming and reasoning text: 497.1 tok/s decode against 567.0 tok/s in llama-bench on the same build. The llama-bench runs are not part of the published raw files.

python3 speed2.py --base-url http://127.0.0.1:34791/v1 --label boole-5090-1411-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/boole-5090-1411-speed-r0/boole-5090-1411-speed-r0/speed2 --contract $RUN/boole-5090-1411-speed-r0/boole-5090-1411-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --sampling {"temperature": 1.0, "top_p": 0.95, "top_k": 20} --sampling-source precision-numerics/Boole-32B-A2.5B-Preview-GGUF (HF @17e92b66, README.md) section 3 (evaluation settings: Temperature 1.0, top-p 0.95, top-k 20, 65,536-token context) and the base model Flint's recommendation (RecursiveSI/flint README: temperature 1.0, top-p 0.95, top-k 20). The author's server command sets no sampling; min_p and penalties are llama-server defaults. --model Boole-32B-A2.5B-Preview-Q2_0.gguf --no-cache-prompt --server-log $HOME/boole-1411/logs/server.log --engine-version-file $RUN/engine-version.txt --server-slots 4 --server-slots-source llama-server default (-np auto, --help: '-np N number of server slots (default: -1, -1 = auto)'; the author's command sets no -np). Server log of the smoke start Nov 15, 2026 (fork 7b225ec3, -c 32768): 'srv load_model: initializing, n_slots = 4, n_ctx_slot = 32768, kv_unified = true' (raw/smoke/server.log) --server-ctx 32768 --server-ctx-source precision-numerics/Boole-32B-A2.5B-Preview-GGUF (HF @17e92b66, README.md) section 6 start command '-c 32768' and boole-demo README ('about 10.8 GB of GPU memory at this 32K context'). Prompt and reply share the window, so 32768-token prompts and the 64K/128K context depths cannot be sent (does_not_fit), the prefill depth 32K is left out of the prefill part for the same reason; nothing is raised by us.

How well it answers

Quality tests on this page ran with the model's own always-on thinking, seed 42, the model card's sampling, and the fixed lists. Answer limits: 30,000 tokens for MMLU-Pro and GPQA, 4,096 for IFEval, 32,768 for LiveCodeBench, 128 for the long-text tasks, 16,384 for GPQA with thinking. A cut-off answer counts as wrong; the model thinks first, so the small limits of IFEval (4,096) and of the long-text tasks (128) leave it little room, and those numbers also measure the token limit. Long text 32K: all 40 prompts have 32,770 tokens and do not fit the 32,768-token window of the card's command (HTTP 400), so the 0.0 % is not a model result. IFEval ran with the card's 4 server slots (4 requests at once, 0 errors). MMLU-Pro, GPQA, the long-text tasks, GPQA with thinking and LiveCodeBench ran with 1 request at a time: with 4 at once the 4 slots share the 32,768-token window, and in a first run 292 requests failed with a context error, so that run is not shown (only its IFEval part, which had no error, is). LiveCodeBench of the first run is not shown either (24 of 40 tasks failed with a context error); it was repeated with 1 request at a time.

The model always thinks (the chat template has no switch). Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-15.

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5090.json --parts mmlupro --job boole-q1-5090-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5090.json --parts gpqa --job boole-q1-5090-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5090.json --parts ifeval --job boole-q-5090-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5090.json --parts lcb --job boole-lcb1-5090-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5090.json --parts ruler --job boole-q1-5090-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5090.json --parts ruler --job boole-q1-5090-1411

bash tools/quality2/measure_all.sh tools/quality2/cards/boole-32b-a2.5b-q2_0-llamacpp-fork-7b225ec3-5090.json --parts think --job boole-q1-5090-1411

Also made this possible

• Model files (Boole-32B-A2.5B-Preview, Q2_0 GGUF, revision 17e92b66)PPrecision Numerics@precision_num Hugging Face
• Base model (Flint)RRecursiveHugging Face
• llama.cpp, which the fork builds onGggml-org@ggml_org GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one machine), driver 610.43.02

Model

Boole-32B-A2.5B-Preview (31.9B MoE, 2.5B active, base Flint) Q2_0 GGUF (precision-numerics/Boole-32B-A2.5B-Preview-GGUF @17e92b66, 8.96 GB, sha256 ea4c8f7f)

Model files (Boole-32B-A2.5B-Preview, Q2_0 GGUF, revision 17e92b66)

huggingface.co/precision-numerics/Boole-32B-A2.5B-Preview-GGUF

Base model (Flint)

huggingface.co/RecursiveSI/flint

Engine

llama.cpp fork of Precision Numerics at commit 7b225ec3, built with CUDA 13.3 — the program that loads the model and answers requests · github.com/PrecisionNumerics/llama.cpp/commit/7b225ec3c9acac279f0748c40136c5bf198acf9d

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   GGML_CUDA_GRAPH_OPT=1 llama-server \
     -m Boole-32B-A2.5B-Preview-Q2_0.gguf \
     -ngl 99 \
     -fa on \
     -ub 1024 \
     -c 32768 \
     --host 127.0.0.1 \
     --port 8888

   What the parts mean

   • -m Boole-32B-A2.5B-Preview-Q2_0.ggufthe model file it loads
   • -c 32768the context window in tokens: how much text the model can see at once
   • -ngl 99how many layers of the model go on the GPU (999 means all of them)
   • -fa onturns on FlashAttention, a faster way to compute attention
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on
   • -ub 1024the most tokens it sends to the GPU in one step (physical batch)

llama.cpp fork PrecisionNumerics/llama.cpp @7b225ec3 (author's pinned commit, build by scripts/setup_llama_cpp.sh, sm_120a), -fa on -ub 1024 -c 32768, GGML_CUDA_GRAPH_OPT=1, thinking always on; GGML_CUDA_GRAPH_OPT=1 llama-server -m Boole-32B-A2.5B-Preview-Q2_0.gguf -ngl 99 -fa on -ub 1024 -c 32768 --host 127.0.0.1 --port 8888 (the command of the model card, unchanged except host and port; fork Precision Numerics/llama.cpp at commit 7b225ec3, built with the author's script scripts/setup_llama_cpp.sh in boole-demo)