Edge0-145B on one RTX 5060 Ti 16 GB: the maker's engine, the 16 GiB + 20 GiB row
Weights: Q3_K (3.44 bits per weight for the gate and up experts, IQ4_NL for the down experts; by Edge0, revision a19f88cc, weight files equal to revision 4136439d)
1x NVIDIA GeForce RTX 5060 Ti 16 GB (16,311 MiB, PCIe 5.0 x8; one machine, second card not used), Intel Core i7-12700K (20 threads, no AVX-512), 31 GiB RAM of which the engine's process group may use 20 GiB with no swap, weights on an NVMe drive, driver 595.71.05, kernel 6.12.86 (Debian 13)
Not ranked yet. The rank needs a finished, verified Gauntlet race (the game test), and this setup has not played it yet. It shows Speed v2, Quality v2 and the memory test, every number with its raw files and exact command. How the game test works.
The maker's own tool first
The maker's benchmark tool (tools/strata_bench.py, 3 runs per test, medians, thinking off, temperature 0) on this card. Columns: chat decode in tok/s, prefill at 4K and at 16K tokens in tok/s, agent decode in tok/s. The maker's row was measured on a 24 GB RTX 5090 D with the VRAM capped to 16 GiB and the RAM capped to 20 GiB; this card is a real 16 GB card, only the limits match (README of the model, section on the capped rows). The "no token" entries are the tool's own wording for an agent session in which no first token was counted; the maker's repository had an issue about this message (Edge0-AI/Edge0, issue 222), which was closed as completed on 16 Nov 2026 at 10:33 UTC (state read on 16 Nov 2026); our runs date from before that. Our standard test, Speed v2, follows below and ran with the engine words of the second variant plus --kv-resident 32768 and --no-prefill-borrow. The gap to the maker's row and what we found about it are explained under the standard test.
| Setup | Chat decode | Prefill 4K | Prefill 16K | Agent decode | Source |
|---|---|---|---|---|---|
| The maker's table row, 16 GiB VRAM + 20 GiB RAM, Q3_K (RTX 5090 D 24 GB with the memory capped) | 36.8 | 249 | 335 | 44.9 | model card (README, table row and the note on the capped card) |
| This card, the bench file's engine words 1:1 (24 expert workers, PCIe fraction 0.1, no resident KV, no prefill borrow) | 4.5 | 59.3 | 59.6 | no token in 3 of 3 sessions | result · data · run · command |
| This card, without --pool-workers 24 and --pcie-frac 0.1 (the maker's advice for smaller CPUs) | 12.6 | 59.3 | 59.1 | no token in 3 of 3 sessions | result · data · run · command |
| This card, as the line above, with --kv-resident 32768 (shipped model.json) and without --no-prefill-borrow | 12.8 | 81.7 | 90.0 | 9.9 in 1 of 3 sessions, 2 sessions no token | result · data · run · command |
How fast it runs
| Measurement | Result |
|---|---|
| Decode | 12.9 tok/s |
| Words per second | 8.9 words/s |
| Characters per second | 62.8 chars/s |
| Decode, code prompt | 13.6 tok/s |
| Decode, prose prompt | 12.1 tok/s |
| Spread of the decode runs (CV) | 4.9 % |
| Spread of the code runs (CV) | 4.9 % |
| Spread of the prose runs (CV) | 4.5 % |
| Decode, window 1 | 12.2 tok/s |
| Prefill at 2K | 58.7 tok/s |
| Prefill at 8K | 59.3 tok/s |
| Prefill at 32K | 59.2 tok/s |
| Time to first token at 2K | 34.9 s |
| Time to first token at 8K | 138.2 s |
| Time to first token at 32K | 553.7 s |
| Draft acceptance rate | 55.6 % |
| 1 request at once (total) | 6.7 tok/s |
| 2 at once (total) | 6.7 tok/s |
| 4 at once (total) | 6.7 tok/s |
| 8 at once (total) | 6.8 tok/s |
| 1 request at once (per request) | 6.7 tok/s |
| 2 at once (per request) | 5.0 tok/s |
| 4 at once (per request) | 2.8 tok/s |
| 8 at once (per request) | 1.5 tok/s |
| Server slots (1: further requests wait in the engine's queue) | 1 slot |
Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once (later supplementary run of 2026-11-16, same setup): C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-16. Raw files: speed result · speed contract · log · speed result · speed contract · log.
Requests at once: the server has 1 slot (from the author's recipe). C2, C4, C8 exceed the slots: the extra requests wait in the queue, and they were measured anyway. What ran. Edge0-145B (a latent sparse MoE with 145B parameters and 10.12B active, built on Qwen3.8-Flash-Next) in the Q3_K files of the maker (Hugging Face Edge0/Edge0-145B-q3k, revision a19f88cc, whose weight files equal revision 4136439d; 96.3 GiB on disk, of which 47.7 GiB are a table that stays on the SSD) on the maker's own engine, Edge0-AI/Edge0 at commit 257ff84, built from source with CUDA 13.3. The context is 131,072 tokens, with one request at a time. Where this run differs from the maker's row. The maker's table row for 16 GiB VRAM + 20 GiB RAM comes from an RTX 5090 D with 24 GB whose VRAM is capped with --vram-reserve-mib 8192 and whose system RAM is capped by a cgroup (MemoryMax=20G, no swap); the maker's README says it is the same card, not a different one. This run uses a real 16 GB card with the same RAM cap (MemoryMax=20G, MemorySwapMax=0). The maker's CPU and drive are not published; this machine has an i7-12700K with 20 threads and AVX2 only, and the weights sit on an NVMe drive in a ZFS pool. Engine words. The standard test below ran with the engine words of the maker's bench file (model-bench.json) without --pool-workers 24 and --pcie-frac 0.1, which an Edge0 team member advised on Hugging Face (15 Nov 2026) for CPUs smaller than the maker's, and with --kv-resident 32768 from the shipped model.json; it keeps --no-prefill-borrow. With the bench file's words 1:1 the maker's tool measured 4.5 tok/s chat decode (24 expert workers on 20 threads; the engine warns about it); the table before the standard test shows all three variants. Against the maker's own figures. The maker's tool is tools/strata_bench.py (3 runs per test, medians with ranges); the maker's row lists 36.8 tok/s chat decode, 249 and 335 tok/s prefill at 4K and 16K and 44.9 tok/s agent decode. We ran the tool on this card in three variants; they are in the table above, before our standard test, with their raw output. Our standard test (Speed v2) is a different method: 512-token answers, temperature 0, seed 42, thinking off, prompts of 2K, 8K and 32K tokens counted by the model's tokenizer. Why this card is slower than the maker's row, as far as we found it: with 24 expert workers on 20 threads the decode was 4.5 tok/s, without them 12.6; the engine reads the expert files through the page cache at about 0.73 GB/s (its own probe; direct reads measured 2.34 GB/s, and the engine chooses the page-cache path because its rule asks for 4 GB/s), the prefill is limited by reading experts from the file tier for every chunk (4K and 16K prompts both give 59 tok/s), and the CPU has no AVX-512. We changed nothing in the system settings. The details of this diagnosis are not part of the raw files. What is missing. The first speed run stopped at the 2-at-once step of the concurrency part: the guard saw a foreign kernel process (kcompactd0, memory compaction, up to 98 % of one core) for two samples in a row after 3 repeats, so the whole 1, 2, 4 and 8 at once series on this page is from a repeat on a freshly started server (its own job). A run for the long-prompt steps (16K, 32K, 64K, 128K) ended at the 3-hour limit of the speed test before the 128K step and was not registered, so this page has no long-prompt table; at this prefill speed the 128K step cannot finish inside the limit. No quality test was run on this card.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 speed2.py --base-url http://127.0.0.1:37791/v1 --label edge0-145b-16gb-1511-a2-2-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $HOME/edge0-1511/raw-1511/B-a2-20261116-004337/measure/raw/ --contract $HOME/edge0-1511/raw-1511/B-a2-20261116-004337/measure/raw/ --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/s128/tok/qwen38-flashnext/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.0} --sampling-source The model card Edge0/Edge0-145B-q3k README.md @4136439d names no sampling. The author's own speed tool sends temperature 0 (edge0-strata/tools/strata_bench.py:133 @7b60076) and the author's docs say identical requests at temperature 0 can differ because decoding is not bit-for-bit repeatable (edge0-infer/docs/edge0-145b.md:131). Seed 42 sent anyway for speed. --model edge0_145b --no-cache-prompt --server-log $HOME/edge0-1511/logs/server.log --engine-version-file $HOME/edge0-1511/raw-1511/B-a2-20261116-004337/measure/logs/engine-version.txt --server-slots 1 --server-slots-source docs: edge0-infer/docs/edge0-145b.md:131 (Edge0-AI/Edge0 @257ff84): 'Concurrent requests wait in edge0-strata's queue instead of receiving 429 as with the native server' = one request at a time; no batch flag in the author's launch (edge0-strata/edge0/models/edge0-145b.json, model-bench.json). --server-ctx 131072 --server-ctx-source edge0-infer/docs/edge0-145b.md:97 (--context default 131072) and edge0-strata/edge0/models/edge0-145b.json (--max-context 131072) @7b60076; HF README 'Context 256K' is the model maximum, not the author's launch value.
# Requests at once: supplementary run of 2026-11-16 (raw folder edge0-145b-16gb-1511-a2-2-conc-speed-r0-conc)
python3 speed2.py --base-url http://127.0.0.1:35065/v1 --label edge0-145b-16gb-1511-a2-2-conc-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $HOME/edge0-1511/raw-1511/B-conc-20261116-023148/measure/raw/ --contract $HOME/edge0-1511/raw-1511/B-conc-20261116-023148/measure/raw/ --parts conc --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/s128/tok/qwen38-flashnext/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.0} --sampling-source The model card Edge0/Edge0-145B-q3k README.md @4136439d names no sampling. The author's own speed tool sends temperature 0 (edge0-strata/tools/strata_bench.py:133 @7b60076) and the author's docs say identical requests at temperature 0 can differ because decoding is not bit-for-bit repeatable (edge0-infer/docs/edge0-145b.md:131). Seed 42 sent anyway for speed. --model edge0_145b --no-cache-prompt --server-log $HOME/edge0-1511/logs/server.log --engine-version-file $HOME/edge0-1511/raw-1511/B-conc-20261116-023148/measure/logs/engine-version.txt --server-slots 1 --server-slots-source docs: edge0-infer/docs/edge0-145b.md:131 (Edge0-AI/Edge0 @257ff84): 'Concurrent requests wait in edge0-strata's queue instead of receiving 429 as with the native server' = one request at a time; no batch flag in the author's launch (edge0-strata/edge0/models/edge0-145b.json, model-bench.json). --server-ctx 131072 --server-ctx-source edge0-infer/docs/edge0-145b.md:97 (--context default 131072) and edge0-strata/edge0/models/edge0-145b.json (--max-context 131072) @7b60076; HF README 'Context 256K' is the model maximum, not the author's launch value. --supplement-of edge0-145b-16gb-1511-a2-2Quality
Not run on this card: no Quality v2 test, so the page shows no quality number. The RTX 5090 pages of this model have quality numbers.
Also made this possible
- Model files (Edge0-145B, Q3_K, revision a19f88cc; the weight files equal revision 4136439d)EEdge0 on Hugging FaceHugging Face
- Base model (Qwen3.8-Flash-Next)
Qwen@Alibaba_Qwen Hugging Face GitHub Website - Strata, which the engine builds onNNiko1221GitHub
- llama.cpp (ggml), part of the engine build
ggml-org@ggml_org GitHub Hugging Face Website
Run it yourself
What you need and how to start it, step by step. The results below come from exactly this setup.
What you need
- Hardware
- 1x NVIDIA GeForce RTX 5060 Ti 16 GB (16,311 MiB, PCIe 5.0 x8; one machine, second card not used), Intel Core i7-12700K (20 threads, no AVX-512), 31 GiB RAM of which the engine's process group may use 20 GiB with no swap, weights on an NVMe drive, driver 595.71.05, kernel 6.12.86 (Debian 13)
- Model
- Edge0-145B (Edge0, 145B latent sparse MoE, 10.12B active, base Qwen3.8-Flash-Next) Q3_K (HF Edge0/Edge0-145B-q3k, weights = rev 4136439d = a19f88cc, pack 96.3 GiB incl. 47.7 GiB PLE table on SSD)
- Model files (Edge0-145B, Q3_K, revision a19f88cc; the weight files equal revision 4136439d)
- huggingface.co/Edge0/Edge0-145B-q3k
- Base model (Qwen3.8-Flash-Next)
- huggingface.co/Qwen/Qwen3.8-Flash-Next
- Engine
- edge0-infer with edge0-strata at commit 257ff84, built from source; the process group capped at 20 GiB of RAM without swap — the program that loads the model and answers requests · github.com/Edge0-AI/Edge0
Steps
Get the engine and the model files from the list above.
Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.
edge0-infer serve \ --model Edge0-145B-q3k \ --device cuda:0 \ --host 127.0.0.1 \ --port 8888 \ --context 131072 \ --strata-arg \ --pcie-mode dma \ --strata-arg \ --vram-reserve-mib 700 \ --strata-arg \ --kv-resident 32768 \ --strata-arg \ --resident-experts \ --strata-arg \ --no-prefill-borrow \ --strata-arg \ --conversation-cache-mib 0 (systemd-run \ --user, MemoryMax=20G, MemorySwapMax=0). These are the engine words of the maker's bench file (model-bench.json) without --pool-workers 24 and --pcie-frac 0.1, with --kv-resident 32768 from the shipped model.json. Engine: Edge0-AI/Edge0 at 257ff84, built from source with CUDA 13.3 (there is no Linux package).What the parts mean
--model Edge0-145B-q3kthe model file it loads--host 127.0.0.1who can connect: only this computer--port 8888the network port the server listens on
Start text for this setupfor reproducing↓
You do not need this to run the model. It is the start text of this setup, based on the setup card of this run, with local paths, machine names and addresses shortened or replaced by placeholders, notes added on some pages, and the engine build in front, so anyone can repeat the result.
Edge0-AI/Edge0 edge0-infer serve + edge0-strata (fork of Niko1221/Strata), source build CUDA 13.3 sm_120, 16 GiB + 20 GiB row words (A-2: model-bench.json without --pool-workers 24 and --pcie-frac 0.1, --kv-resident 32768), cgroup MemoryMax=20G; edge0-infer serve --model Edge0-145B-q3k --device cuda:0 --host 127.0.0.1 --port 8888 --context 131072 --strata-arg --pcie-mode dma --strata-arg --vram-reserve-mib 700 --strata-arg --kv-resident 32768 --strata-arg --resident-experts --strata-arg --no-prefill-borrow --strata-arg --conversation-cache-mib 0 (systemd-run --user, MemoryMax=20G, MemorySwapMax=0). These are the engine words of the maker's bench file (model-bench.json) without --pool-workers 24 and --pcie-frac 0.1, with --kv-resident 32768 from the shipped model.json. Engine: Edge0-AI/Edge0 at 257ff84, built from source with CUDA 13.3 (there is no Linux package).Context and sampling: 131,072 tokens (the maker's default, --context 131072), 1 request at a time (further requests wait in the engine's queue), int8 KV cache with 32,768 cells resident on the card (the value of the shipped model.json), speculative decoding on, thinking off per request in the speed test, temperature 0 and seed 42 sent with every request. The server process group is capped at 20 GiB of RAM with no swap, as in the maker's 16 GiB + 20 GiB row.. Hardware: 1x NVIDIA GeForce RTX 5060 Ti 16 GB (16,311 MiB, PCIe 5.0 x8; one machine, second card not used), Intel Core i7-12700K (20 threads, no AVX-512), 31 GiB RAM of which the engine's process group may use 20 GiB with no swap, weights on an NVMe drive, driver 595.71.05, kernel 6.12.86 (Debian 13). All raw files of this setup: raw/v2/edge0-145b-q3k-5060ti-edge0-infer-257ff84.