Mia's Nemotron 3.5 Bolt recipe on one RTX 5090: NVIDIA Nemotron 3.5 Bolt 74B-A9B

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-02. Raw files: speed result · speed contract · log · long answer.

Requests at once: the server has 2 slots (PARALLEL=2 instead of the recipe's 16, see the changes below). C4, C8 exceed the slots: the extra requests wait in the queue, and they were measured anyway. The speed test ran with Mia's recipe (MiaAI-Lab/Nemotron-3.5-Bolt-TensorFold-DGX-Spark, commit 6b064a31 of 1 Nov 2026, the head of main on 2 Nov 2026) on one RTX 5090 instead of a DGX Spark. Her recipe is written for the DGX Spark (GB10, ARM); these changes were forced by the 5090 and are listed with their reasons: the engine (TensorFold v1.1.6, commit eea9dc9c) and Mia's patch 0002 are built for sm_120 (her patch and build call target sm_121); the Docker build uses the x86_64 Zig compiler instead of the ARM one; the video-input library (libtfvideo) does not link on x86_64 and was left out, so this build has no video input (text and image input are unchanged); the memory reserves are TENSORFOLD_MEMORY_RESERVE_GIB=1 instead of 10 and BOLT_KV_RESERVE_GIB=0.25 instead of 3 (her floors protect the shared memory of the GB10 box; on the 5090 the weights of 22.4 GiB would not fit beside them); PARALLEL=2 instead of 16, PROMPT_CACHE_GIB=1 instead of 8 and VISION=0 instead of 1 (the 32 GB card has room for the weights and a small KV cache only). A first session with 8 streams (job bolt5090-mia-0211-r2) ran into the open issue TensorFold #1654 / Mia #4 (HTTP 500 from the Bolt forward pass under sustained load with 2 or more parallel clients) in the quality tests; this page shows the repeat with 2 streams (job r3), where every step answered. The server has 2 slots, which means the levels of 4 and 8 requests at once wait in the queue and their per-request numbers include the waiting. The container's name and port and the removal of a build-folder cleanup (my rule: no deletion) change no number. Sampling is the model card's: temperature 1.0, top_p 0.95; seed 42; the request flag for no thinking; no cached tokens. The headline 242.67 tok/s is the mean of the code prompt (219.84) and the prose prompt (265.50), medians of two runs each (spread 10.5 % and 2.4 %). Drafts were accepted at a rate of 32.0 %. This is a different machine from Mia's (her numbers are measured on a DGX Spark), so her numbers are not a reference for this page.

python3 speed2.py --base-url http://127.0.0.1:46741/v1 --label bolt5090-mia-0211-r3-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/bolt5090-mia-0211-r3-speed-r0/bolt5090-mia-0211-r3-speed-r0/speed2 --contract $RUN/bolt5090-mia-0211-r3-speed-r0/bolt5090-mia-0211-r3-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/.cache/huggingface/hub/models--nvidia--NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB/snapshots/a326822af111290f2af5ac59ef4b5040347dabc4/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 1.0, "top_p": 0.95} --sampling-source https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB README.md line 66 ('Text: temperature 1.0, top_p 0.95') @a326822a; local copy source/hf-bolt-24gb-README.md; Mia scripts/config.sh:90-93 --model Nemotron-3.5-Bolt --no-cache-prompt --server-log $RUN/server-A.log --engine-version-file $RUN/engine-version.txt --server-slots 2 --server-slots-source Mia scripts/config.sh:59-61 (PARALLEL 1 to 16, default 16); lowered to 2 on 2 Nov 2026: the open issue TensorFold #1654 / Mia #4 (HTTP 500 from the Bolt forward pass with 2 or more parallel clients under sustained load) hit run r2 with 8 slots (its IFEval, long-text, LiveCodeBench and thinking parts were invalid); with fewer streams more video memory stays free for the KV cache (128K). C4 and C8 then wait in the queue. --server-ctx 262144 --server-ctx-source card box-6-bolt74b-iq2-lilicorr-tensorfold-mia-eea9dc9-sm120-5090-p2 field ctx

How well it answers

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-11-02.

Note from the run: Quality ran on this card on 2 Nov 2026 in one session (job bolt5090-mia-0211-r3). The questions of each test ran 2 at a time, the slot count of the server (same questions, limits and seed 42 for every question); the median time per question of the thinking test (58.7 s) was measured under that load. MMLU-Pro and GPQA Diamond ran with an answer limit of 48 tokens (the fixed method asks for the letter only), IFEval with 4,096, LiveCodeBench with 32,768, the long-text tests with 128 tokens. A cut-off answer counts as wrong: 3 of 200 IFEval answers, 4 of 40 LiveCodeBench answers and 10 of 30 GPQA answers with thinking (limit 16,384 tokens, median 11,807 thinking tokens) were cut off; no answer of the other tests was. GPQA Diamond ran from the public mirror fingertap/GPQA-Diamond (198 questions), because the original is gated. Thinking was off in all tests except GPQA with thinking. The first session with 8 streams is named in the speed note above and is not shown; these are my own tests, and no other run is mixed into the quality numbers on this page. The answers file of the thinking test (GPQA 30) is not published: the site's automatic filter for private names flagged a word in the model's reasoning text. The row was counted from the complete file.

bash tools/quality2/measure_all.sh tools/quality2/cards/bolt74b-iq2-lilicorr-tensorfold-mia-eea9dc9-sm120-5090-p2.json --parts mmlupro --job bolt5090-mia-0211-r3

bash tools/quality2/measure_all.sh tools/quality2/cards/bolt74b-iq2-lilicorr-tensorfold-mia-eea9dc9-sm120-5090-p2.json --parts gpqa --job bolt5090-mia-0211-r3

bash tools/quality2/measure_all.sh tools/quality2/cards/bolt74b-iq2-lilicorr-tensorfold-mia-eea9dc9-sm120-5090-p2.json --parts ifeval --job bolt5090-mia-0211-r3

bash tools/quality2/measure_all.sh tools/quality2/cards/bolt74b-iq2-lilicorr-tensorfold-mia-eea9dc9-sm120-5090-p2.json --parts lcb --job bolt5090-mia-0211-r3

bash tools/quality2/measure_all.sh tools/quality2/cards/bolt74b-iq2-lilicorr-tensorfold-mia-eea9dc9-sm120-5090-p2.json --parts ruler --job bolt5090-mia-0211-r3

bash tools/quality2/measure_all.sh tools/quality2/cards/bolt74b-iq2-lilicorr-tensorfold-mia-eea9dc9-sm120-5090-p2.json --parts ruler --job bolt5090-mia-0211-r3

bash tools/quality2/measure_all.sh tools/quality2/cards/bolt74b-iq2-lilicorr-tensorfold-mia-eea9dc9-sm120-5090-p2.json --parts think --job bolt5090-mia-0211-r3

How much context fits

Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again, one prompt that fills about 95 % of that size is sent, and the model must answer. On this page the recipe fixes the context at 262,144 tokens, so every start used that context (see the note below). The largest size that loaded and answered counts. Decode speed is measured on that answer at the largest size. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. Measured 2026-11-02. Raw files: result of every step · log · contract.

Note from the run: The recipe fixes the context (262,144 tokens, 2 streams here): for each size from 4K up to 256K the server was started once with that context, a prompt of about 95 % of the size was sent, and the model answered. Decode speed at these sizes is not measured (the server returns no timings). The memory number is the highest GPU memory in use during the largest step (nvidia-smi memory.used, absolute, including the weights).

bash tools/quality2/measure_all.sh tools/quality2/cards/bolt74b-iq2-lilicorr-tensorfold-mia-eea9dc9-sm120-5090-p2.json --parts fit --job bolt5090-mia-0211-r3

Also made this possible

• Model (24 GB pack, revision a326822a)NVIDIAHugging Face GitHub Website
• Draft model (LiLiCorr, revision 4c6a5613)NVIDIAHugging Face GitHub Website
• Engine (TensorFold v1.1.6, commit eea9dc9c)Ash Hart@ashxhart GitHub Hugging Face Website

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB, one GPU. Driver 610.43.02, kernel 6.12.95.

Model

nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB @a326822a + nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr @4c6a5613

Model (24 GB pack, revision a326822a)

huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB

Draft model (LiLiCorr, revision 4c6a5613)

huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr

Engine

TensorFold 1.1.6 built for the RTX 5090 (sm_120), 24 GB pack with IQ2_XXS experts and NVFP4 Mamba layers, LiLiCorr drafts, 262,144 tokens of context, 2 streams — the program that loads the model and answers requests · github.com/ashhart/TensorFoldBuild and settings we used: TensorFold v1.1.6 tensorfold-native (Zig, CUDA, sm_120) + Mia Patch 0002 + libbolt (sm_120a), LiLiCorr drafts, built from source in Docker

Recipe (TensorFold on a DGX Spark), commit 6b064a31, ported to the RTX 5090 with the changes listed on this page

github.com/MiaAI-Lab/Nemotron-3.5-Bolt-TensorFold-DGX-Spark by Mia's AI Lab

Steps

1. Get the recipe, the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8895.

   Copy

   TENSORFOLD_MEMORY_RESERVE_GIB=1 \
     BOLT_KV_RESERVE_GIB=0.25 \
     PORT=8895 \
     HOST=127.0.0.1 \
     PARALLEL=2 \
     PROMPT_CACHE_GIB=1 \
     VISION=0 ./start.sh

   What the parts mean

   • PORT=8895the network port the server listens on

TensorFold v1.1.6 tensorfold-native (Zig, CUDA, sm_120) + Mia Patch 0002 + libbolt (sm_120a), LiLiCorr drafts, built from source in Docker; TENSORFOLD_MEMORY_RESERVE_GIB=1 BOLT_KV_RESERVE_GIB=0.25 PORT=8895 HOST=127.0.0.1 PARALLEL=2 PROMPT_CACHE_GIB=1 VISION=0 ./start.sh