Recursive's Flint 40B on a 16 GB card: Unsloth UD-Q2_K_XL GGUF

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-02. Raw files: speed result · speed contract · log · long answer.

The speed test ran with the maker's server flags (README of the NVFP4 repo: one slot, flash attention, F16 cache, the maker's sampling, no speculation; seed 42 is our setting), changed only where the 16 GB card forces it: Unsloth's UD-Q2_K_XL GGUF (13,562,207,680 bytes = 12.63 GiB, about 2.7 bits per weight, our calculation: file size times 8 divided by the 39.8 billion parameters) instead of the maker's NVFP4 file (22.9 GB), and a context of 65,536 tokens instead of 262,144. Added by me, with no influence on the numbers: -fit off, --cache-ram 0, --no-webui, --metrics, the port and a 14 GB container memory limit. The engine is llama.cpp b11972 (commit cf24317bea, built from source, CUDA 13.3, sm_120a). The headline 115.89 tok/s is the mean of the code prompt (116.01) and the prose prompt (115.77), medians of two runs each (spread 0.06 % and 0.08 %). Flint has no switch for thinking: its reasoning is always on (the maker's guide says so, and the chat template has no thinking flag). My tests send the usual request flag for no thinking, but Flint ignores it and starts every reply with its reasoning. Because every reply of the 512-token runs is reasoning text, the visible words per second are 0, so no words-per-second values and no "500 words" estimate are shown. The server has one slot, so 2, 4 and 8 requests at once wait in line; the total stays between 99.3 and 99.5 tok/s and the per-request numbers include the waiting. 64K and 128K prompts did not fit the server context of 65,536 tokens, so the long-prompt table has rows for 16K and 32K only. The page of the RTX 5090 run of this model (NVFP4 file, 262,144 tokens of context, llama.cpp b11803) is another setup, so its numbers are not a like-for-like comparison. Other processes used up to 100 % CPU during the speed test (the speed contract names the busiest one).

python3 speed2.py --base-url http://127.0.0.1:41385/v1 --label flint16-0211-q2kxl-speed-r1 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/flint16-0211-q2kxl-speed-r1/flint16-0211-q2kxl-speed-r1/speed2 --contract $RUN/flint16-0211-q2kxl-speed-r1/flint16-0211-q2kxl-speed-r1/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --sampling {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0} --sampling-source RecursiveSI/flint sglang/README.md:159-161 'Sampling: temperature 1.0, top_p 0.95, top_k 20 ... Greedy decoding is not validated' @2a247be0c0, README.md:86-89, generation_config.json:11-13; min_p 0 from RecursiveSI/flint-int4-gguf README.md:35 (--min-p 0) @9c47151. Flint has no non-thinking mode, so the instruct and thinking profiles are identical (copies in $HOME/engines/llamacpp-flint-2110/author-files-2a247be/ and $HOME/models/flint-40b-q4_1-9c47151/author-files/) --model flint --no-cache-prompt --server-log $HOME/flint-0211/logs/server.log --engine-version-file $RUN/engine-version.txt --server-slots 1 --server-slots-source RecursiveSI/flint-nvfp4-gguf README.md:30 (-np 1) @6ec43e19 (llama-server command README.md:29-33), copy $HOME/bench/nvfp4-README.md:30 --server-ctx 65536 --server-ctx-source card box-5-flint-40b-udq2kxl-llamacpp-b11972-5060ti field ctx

Quality

Quality tests were not run on this setup. Flint has no switch for thinking and reasons before every answer. The two fixed lists that allow 48 tokens per answer (MMLU-Pro and GPQA Diamond, one letter each) cannot measure such a model: on the RTX 5090 page of this model 369 of 400 MMLU-Pro answers and all 198 GPQA Diamond answers were cut off and count as wrong. No quality numbers are shown here, and this setup has no rank.

How much context fits

Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed is measured on that answer at the largest size. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. First size that failed: 96K (98,304 tokens), it did not come up: not up after 900 s. Measured 2026-11-02. Raw files: result of every step · log · contract.

Note from the run: The largest size tested is 65,536 tokens, the size the server was started with for the other tests; it is not necessarily the largest the card could hold. Every size from 4K to 64K loaded and answered. The next size, 96K (98,304 tokens), did not come up within 900 s, so the maker's 262,144 tokens were not tried. The memory number is the GPU memory in use for the whole card (the weights are 12.63 GiB and the F16 cache grows with the context). The maker's README asks for at least 48 GB of GPU memory.

bash tools/quality2/measure_all.sh tools/quality2/cards/flint-40b-udq2kxl-llamacpp-b11972-5060ti.json --parts fit --job flint16-0211-q2kxl

Also made this possible

• Quant (UD-Q2_K_XL GGUF, revision 20468edf)Unsloth AIHugging Face @unslothai GitHub Website
• Engine (llama.cpp b11972, commit cf24317bea)ggml-org (llama.cpp)GitHub @ggml_org Hugging Face Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5060 Ti 16 GB (one GPU, the whole model in video memory). Driver 595.71.05, kernel 6.12.86.

Model

Recursive Flint 40B (39.8B MoE, ~2.5B active) UD-Q2_K_XL GGUF (unsloth/flint-GGUF @20468edf)

Model and recipe (the maker's llama-server flags)

huggingface.co/RecursiveSI/flint-nvfp4-gguf

Quant (UD-Q2_K_XL GGUF, revision 20468edf)

huggingface.co/unsloth/flint-GGUF

Engine

llama.cpp b11972, one RTX 5060 Ti 16 GB, the maker's llama-server flags with a smaller quant and a 65,536-token context — the program that loads the model and answers requests · github.com/ggml-org/llama.cpp/releases/tag/b11972Build and settings we used: llama.cpp b11972 @cf24317bea (source build, CUDA 13.3 container nvcr.io/nvidia/pytorch:26.07-py3, sm_120a), F16 KV, no speculation

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   docker run \
     --gpus device=<GPU1 uuid> llama-server \
     -m flint-UD-Q2_K_XL.gguf \
     --alias flint \
     -ngl all \
     -c 65536 \
     -np 1 \
     -fa on \
     -ctk f16 \
     -ctv f16 \
     --jinja \
     --temp 1 \
     --top-p 0.95 \
     --top-k 20 \
     --min-p 0 \
     --host 127.0.0.1 \
     --port 8888 \
     -fit off \
     --cache-ram 0 \
     --no-webui \
     --metrics

   What the parts mean

   • -m flint-UD-Q2_K_XL.ggufthe model file it loads
   • -c 65536the context window in tokens: how much text the model can see at once
   • -np 1how many requests it answers at the same time
   • --jinjauses the chat template that comes with the model
   • -fa onturns on FlashAttention, a faster way to compute attention
   • --temp 1sampling temperature: how random the answers are
   • --top-p 0.95top-p sampling: only the most likely next tokens that together reach this probability are considered
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on
   • --cache-ram 0the most system RAM, in MiB, used to keep earlier prompts for reuse (0 turns this off)
   • -fit offturns off llama.cpp's automatic adjustment of unset settings to the free GPU memory; the command's settings are used as given
   • --no-webuiturns off the built-in chat web page; only the API answers
   • --metricsturns on a /metrics page with server statistics (Prometheus format)
   • --alias flintthe model name clients use in their requests

llama.cpp b11972 @cf24317bea (source build, CUDA 13.3 container nvcr.io/nvidia/pytorch:26.07-py3, sm_120a), F16 KV, no speculation; docker run --gpus device=<GPU1 uuid> llama-server -m flint-UD-Q2_K_XL.gguf --alias flint -ngl all -c 65536 -np 1 -fa on -ctk f16 -ctv f16 --jinja --temp 1 --top-p 0.95 --top-k 20 --min-p 0 --host 127.0.0.1 --port 8888 -fit off --cache-ram 0 --no-webui --metrics