Home / 16 GB

16 GB · RTX 5060 Ti 16 GB

Edge0-145B on one RTX 5060 Ti 16 GB: the maker's engine, the 16 GiB + 20 GiB row

Weights: Q3_K (3.44 bits per weight for the gate and up experts, IQ4_NL for the down experts; by Edge0, revision a19f88cc, weight files equal to revision 4136439d)

1x NVIDIA GeForce RTX 5060 Ti 16 GB (16,311 MiB, PCIe 5.0 x8; one machine, second card not used), Intel Core i7-12700K (20 threads, no AVX-512), 31 GiB RAM of which the engine's process group may use 20 GiB with no swap, weights on an NVMe drive, driver 595.71.05, kernel 6.12.86 (Debian 13)

Engine (edge0-infer, edge0-strata)Edge0
Edge0-AI/Edge0 edge0-infer serve + edge0-strata (fork of Niko1221/Strata), source build CUDA 13.3 sm_120, 16 GiB + 20 GiB row words (A-2: model-bench.json without --pool-workers 24 and --pcie-frac 0.1, --kv-resident 32768), cgroup MemoryMax=20Gcommit Edge0-AI/Edge0 257ff84ae8a89dbf34239dcf8803c91d07df94b8; ggml llama.cpp 3cf03257f219afbe7334045ff7c6a06ac68c627d; weights HF Edge0/Edge0-145B-q3k (rev 4136439d, files equal to a19f88cc)1x RTX 5060 Ti 16 GB (the storage box GPU0)measured 2026-11-16

Not ranked yet. The rank needs a finished, verified Gauntlet race (the game test), and this setup has not played it yet. It shows Speed v2, Quality v2 and the memory test, every number with its raw files and exact command. How the game test works.

Speed, the author's method

The maker's own tool first

The maker's benchmark tool (tools/strata_bench.py, 3 runs per test, medians, thinking off, temperature 0) on this card. Columns: chat decode in tok/s, prefill at 4K and at 16K tokens in tok/s, agent decode in tok/s. The maker's row was measured on a 24 GB RTX 5090 D with the VRAM capped to 16 GiB and the RAM capped to 20 GiB; this card is a real 16 GB card, only the limits match (README of the model, section on the capped rows). The "no token" entries are the tool's own wording for an agent session in which no first token was counted; the maker's repository had an issue about this message (Edge0-AI/Edge0, issue 222), which was closed as completed on 16 Nov 2026 at 10:33 UTC (state read on 16 Nov 2026); our runs date from before that. Our standard test, Speed v2, follows below and ran with the engine words of the second variant plus --kv-resident 32768 and --no-prefill-borrow. The gap to the maker's row and what we found about it are explained under the standard test.

SetupChat decodePrefill 4KPrefill 16KAgent decodeSource
The maker's table row, 16 GiB VRAM + 20 GiB RAM, Q3_K (RTX 5090 D 24 GB with the memory capped)36.824933544.9model card (README, table row and the note on the capped card)
This card, the bench file's engine words 1:1 (24 expert workers, PCIe fraction 0.1, no resident KV, no prefill borrow)4.559.359.6no token in 3 of 3 sessionsresult · data · run · command
This card, without --pool-workers 24 and --pcie-frac 0.1 (the maker's advice for smaller CPUs)12.659.359.1no token in 3 of 3 sessionsresult · data · run · command
This card, as the line above, with --kv-resident 32768 (shipped model.json) and without --no-prefill-borrow12.881.790.09.9 in 1 of 3 sessions, 2 sessions no tokenresult · data · run · command
Speed v2

How fast it runs

Decode
12.9tok/s
Decode, code
13.6tok/s
Decode, prose
12.1tok/s
500 words
≈ 56.1s
Prefill at 8K
59.3tok/s
First token at 2K
34.9s
MeasurementResult
Decode12.9 tok/s
Words per second8.9 words/s
Characters per second62.8 chars/s
Decode, code prompt13.6 tok/s
Decode, prose prompt12.1 tok/s
Spread of the decode runs (CV)4.9 %
Spread of the code runs (CV)4.9 %
Spread of the prose runs (CV)4.5 %
Decode, window 112.2 tok/s
Prefill at 2K58.7 tok/s
Prefill at 8K59.3 tok/s
Prefill at 32K59.2 tok/s
Time to first token at 2K34.9 s
Time to first token at 8K138.2 s
Time to first token at 32K553.7 s
Draft acceptance rate55.6 %
1 request at once (total)6.7 tok/s
2 at once (total)6.7 tok/s
4 at once (total)6.7 tok/s
8 at once (total)6.8 tok/s
1 request at once (per request)6.7 tok/s
2 at once (per request)5.0 tok/s
4 at once (per request)2.8 tok/s
8 at once (per request)1.5 tok/s
Server slots (1: further requests wait in the engine's queue)1 slot

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once (later supplementary run of 2026-11-16, same setup): C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-16. Raw files: speed result · speed contract · log · speed result · speed contract · log.

Requests at once: the server has 1 slot (from the author's recipe). C2, C4, C8 exceed the slots: the extra requests wait in the queue, and they were measured anyway. What ran. Edge0-145B (a latent sparse MoE with 145B parameters and 10.12B active, built on Qwen3.8-Flash-Next) in the Q3_K files of the maker (Hugging Face Edge0/Edge0-145B-q3k, revision a19f88cc, whose weight files equal revision 4136439d; 96.3 GiB on disk, of which 47.7 GiB are a table that stays on the SSD) on the maker's own engine, Edge0-AI/Edge0 at commit 257ff84, built from source with CUDA 13.3. The context is 131,072 tokens, with one request at a time. Where this run differs from the maker's row. The maker's table row for 16 GiB VRAM + 20 GiB RAM comes from an RTX 5090 D with 24 GB whose VRAM is capped with --vram-reserve-mib 8192 and whose system RAM is capped by a cgroup (MemoryMax=20G, no swap); the maker's README says it is the same card, not a different one. This run uses a real 16 GB card with the same RAM cap (MemoryMax=20G, MemorySwapMax=0). The maker's CPU and drive are not published; this machine has an i7-12700K with 20 threads and AVX2 only, and the weights sit on an NVMe drive in a ZFS pool. Engine words. The standard test below ran with the engine words of the maker's bench file (model-bench.json) without --pool-workers 24 and --pcie-frac 0.1, which an Edge0 team member advised on Hugging Face (15 Nov 2026) for CPUs smaller than the maker's, and with --kv-resident 32768 from the shipped model.json; it keeps --no-prefill-borrow. With the bench file's words 1:1 the maker's tool measured 4.5 tok/s chat decode (24 expert workers on 20 threads; the engine warns about it); the table before the standard test shows all three variants. Against the maker's own figures. The maker's tool is tools/strata_bench.py (3 runs per test, medians with ranges); the maker's row lists 36.8 tok/s chat decode, 249 and 335 tok/s prefill at 4K and 16K and 44.9 tok/s agent decode. We ran the tool on this card in three variants; they are in the table above, before our standard test, with their raw output. Our standard test (Speed v2) is a different method: 512-token answers, temperature 0, seed 42, thinking off, prompts of 2K, 8K and 32K tokens counted by the model's tokenizer. Why this card is slower than the maker's row, as far as we found it: with 24 expert workers on 20 threads the decode was 4.5 tok/s, without them 12.6; the engine reads the expert files through the page cache at about 0.73 GB/s (its own probe; direct reads measured 2.34 GB/s, and the engine chooses the page-cache path because its rule asks for 4 GB/s), the prefill is limited by reading experts from the file tier for every chunk (4K and 16K prompts both give 59 tok/s), and the CPU has no AVX-512. We changed nothing in the system settings. The details of this diagnosis are not part of the raw files. What is missing. The first speed run stopped at the 2-at-once step of the concurrency part: the guard saw a foreign kernel process (kcompactd0, memory compaction, up to 98 % of one core) for two samples in a row after 3 repeats, so the whole 1, 2, 4 and 8 at once series on this page is from a repeat on a freshly started server (its own job). A run for the long-prompt steps (16K, 32K, 64K, 128K) ended at the 3-hour limit of the speed test before the 128K step and was not registered, so this page has no long-prompt table; at this prefill speed the 128K step cannot finish inside the limit. No quality test was run on this card.

Exact command we measured withfor reproducing↓

You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.

Speed v2 (run edge0-145b-16gb-1511-a2-2-speed-r0)
python3 speed2.py --base-url http://127.0.0.1:37791/v1 --label edge0-145b-16gb-1511-a2-2-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $HOME/edge0-1511/raw-1511/B-a2-20261116-004337/measure/raw/ --contract $HOME/edge0-1511/raw-1511/B-a2-20261116-004337/measure/raw/ --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/s128/tok/qwen38-flashnext/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.0} --sampling-source The model card Edge0/Edge0-145B-q3k README.md @4136439d names no sampling. The author's own speed tool sends temperature 0 (edge0-strata/tools/strata_bench.py:133 @7b60076) and the author's docs say identical requests at temperature 0 can differ because decoding is not bit-for-bit repeatable (edge0-infer/docs/edge0-145b.md:131). Seed 42 sent anyway for speed. --model edge0_145b --no-cache-prompt --server-log $HOME/edge0-1511/logs/server.log --engine-version-file $HOME/edge0-1511/raw-1511/B-a2-20261116-004337/measure/logs/engine-version.txt --server-slots 1 --server-slots-source docs: edge0-infer/docs/edge0-145b.md:131 (Edge0-AI/Edge0 @257ff84): 'Concurrent requests wait in edge0-strata's queue instead of receiving 429 as with the native server' = one request at a time; no batch flag in the author's launch (edge0-strata/edge0/models/edge0-145b.json, model-bench.json). --server-ctx 131072 --server-ctx-source edge0-infer/docs/edge0-145b.md:97 (--context default 131072) and edge0-strata/edge0/models/edge0-145b.json (--max-context 131072) @7b60076; HF README 'Context 256K' is the model maximum, not the author's launch value.

# Requests at once: supplementary run of 2026-11-16 (raw folder edge0-145b-16gb-1511-a2-2-conc-speed-r0-conc)
python3 speed2.py --base-url http://127.0.0.1:35065/v1 --label edge0-145b-16gb-1511-a2-2-conc-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $HOME/edge0-1511/raw-1511/B-conc-20261116-023148/measure/raw/ --contract $HOME/edge0-1511/raw-1511/B-conc-20261116-023148/measure/raw/ --parts conc --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/s128/tok/qwen38-flashnext/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.0} --sampling-source The model card Edge0/Edge0-145B-q3k README.md @4136439d names no sampling. The author's own speed tool sends temperature 0 (edge0-strata/tools/strata_bench.py:133 @7b60076) and the author's docs say identical requests at temperature 0 can differ because decoding is not bit-for-bit repeatable (edge0-infer/docs/edge0-145b.md:131). Seed 42 sent anyway for speed. --model edge0_145b --no-cache-prompt --server-log $HOME/edge0-1511/logs/server.log --engine-version-file $HOME/edge0-1511/raw-1511/B-conc-20261116-023148/measure/logs/engine-version.txt --server-slots 1 --server-slots-source docs: edge0-infer/docs/edge0-145b.md:131 (Edge0-AI/Edge0 @257ff84): 'Concurrent requests wait in edge0-strata's queue instead of receiving 429 as with the native server' = one request at a time; no batch flag in the author's launch (edge0-strata/edge0/models/edge0-145b.json, model-bench.json). --server-ctx 131072 --server-ctx-source edge0-infer/docs/edge0-145b.md:97 (--context default 131072) and edge0-strata/edge0/models/edge0-145b.json (--max-context 131072) @7b60076; HF README 'Context 256K' is the model maximum, not the author's launch value. --supplement-of edge0-145b-16gb-1511-a2-2
Quality v2

Quality

Not run on this card: no Quality v2 test, so the page shows no quality number. The RTX 5090 pages of this model have quality numbers.

Credits

Also made this possible

Setup

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware
1x NVIDIA GeForce RTX 5060 Ti 16 GB (16,311 MiB, PCIe 5.0 x8; one machine, second card not used), Intel Core i7-12700K (20 threads, no AVX-512), 31 GiB RAM of which the engine's process group may use 20 GiB with no swap, weights on an NVMe drive, driver 595.71.05, kernel 6.12.86 (Debian 13)
Model
Edge0-145B (Edge0, 145B latent sparse MoE, 10.12B active, base Qwen3.8-Flash-Next) Q3_K (HF Edge0/Edge0-145B-q3k, weights = rev 4136439d = a19f88cc, pack 96.3 GiB incl. 47.7 GiB PLE table on SSD)
Model files (Edge0-145B, Q3_K, revision a19f88cc; the weight files equal revision 4136439d)
huggingface.co/Edge0/Edge0-145B-q3k
Base model (Qwen3.8-Flash-Next)
huggingface.co/Qwen/Qwen3.8-Flash-Next
Engine
edge0-infer with edge0-strata at commit 257ff84, built from source; the process group capped at 20 GiB of RAM without swap — the program that loads the model and answers requests · github.com/Edge0-AI/Edge0

Steps

  1. Get the engine and the model files from the list above.

  2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

    edge0-infer serve \
      --model Edge0-145B-q3k \
      --device cuda:0 \
      --host 127.0.0.1 \
      --port 8888 \
      --context 131072 \
      --strata-arg \
      --pcie-mode dma \
      --strata-arg \
      --vram-reserve-mib 700 \
      --strata-arg \
      --kv-resident 32768 \
      --strata-arg \
      --resident-experts \
      --strata-arg \
      --no-prefill-borrow \
      --strata-arg \
      --conversation-cache-mib 0 (systemd-run \
      --user, MemoryMax=20G, MemorySwapMax=0). These are the engine words of the maker's bench file (model-bench.json) without --pool-workers 24 and --pcie-frac 0.1, with --kv-resident 32768 from the shipped model.json. Engine: Edge0-AI/Edge0 at 257ff84, built from source with CUDA 13.3 (there is no Linux package).

    What the parts mean

    • --model Edge0-145B-q3kthe model file it loads
    • --host 127.0.0.1who can connect: only this computer
    • --port 8888the network port the server listens on
Start text for this setupfor reproducing↓

You do not need this to run the model. It is the start text of this setup, based on the setup card of this run, with local paths, machine names and addresses shortened or replaced by placeholders, notes added on some pages, and the engine build in front, so anyone can repeat the result.

Start, based on the setup card of this run
Edge0-AI/Edge0 edge0-infer serve + edge0-strata (fork of Niko1221/Strata), source build CUDA 13.3 sm_120, 16 GiB + 20 GiB row words (A-2: model-bench.json without --pool-workers 24 and --pcie-frac 0.1, --kv-resident 32768), cgroup MemoryMax=20G; edge0-infer serve --model Edge0-145B-q3k --device cuda:0 --host 127.0.0.1 --port 8888 --context 131072 --strata-arg --pcie-mode dma --strata-arg --vram-reserve-mib 700 --strata-arg --kv-resident 32768 --strata-arg --resident-experts --strata-arg --no-prefill-borrow --strata-arg --conversation-cache-mib 0 (systemd-run --user, MemoryMax=20G, MemorySwapMax=0). These are the engine words of the maker's bench file (model-bench.json) without --pool-workers 24 and --pcie-frac 0.1, with --kv-resident 32768 from the shipped model.json. Engine: Edge0-AI/Edge0 at 257ff84, built from source with CUDA 13.3 (there is no Linux package).

Context and sampling: 131,072 tokens (the maker's default, --context 131072), 1 request at a time (further requests wait in the engine's queue), int8 KV cache with 32,768 cells resident on the card (the value of the shipped model.json), speculative decoding on, thinking off per request in the speed test, temperature 0 and seed 42 sent with every request. The server process group is capped at 20 GiB of RAM with no swap, as in the maker's 16 GiB + 20 GiB row.. Hardware: 1x NVIDIA GeForce RTX 5060 Ti 16 GB (16,311 MiB, PCIe 5.0 x8; one machine, second card not used), Intel Core i7-12700K (20 threads, no AVX-512), 31 GiB RAM of which the engine's process group may use 20 GiB with no swap, weights on an NVMe drive, driver 595.71.05, kernel 6.12.86 (Debian 13). All raw files of this setup: raw/v2/edge0-145b-q3k-5060ti-edge0-infer-257ff84.