Edge0-145B on one RTX 5090: the maker's engine, README start

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-15. Raw files: speed result · speed contract · log · long answer.

What ran. Edge0-145B (a latent sparse MoE with 145B parameters and 10.12B active, built on Qwen3.8-Flash-Next) in the Q3_K files of the maker (Hugging Face Edge0/Edge0-145B-q3k, revision 4136439d; 98.1 GiB on disk, of which 47.7 GiB are a table that stays on the SSD) on the maker's own engine: Edge0-AI/Edge0 at commit 7b60076 (edge0-infer serve with edge0-strata). The engine is built from source with CUDA 13.3 for the RTX 5090, as the maker describes for Linux; there is no Linux package. The context is 131,072 tokens, the maker's default, with one request at a time (further requests wait in the engine's queue). Where this run differs from the maker's setup. (1) Card: the maker's published numbers come from an RTX 5090 D with 24 GB; this run used an RTX 5090 with 32 GB. Nothing was lowered. (2) Build: only the target edge0-infer was built, because the build of the other targets stopped on a compiler test (the maker's pull request 221 is about it). (3) The file latent_dense_q4.bin was not loaded; it is only needed for the du=q4 setting. (4) Prompt lengths in the speed test are counted with the tokenizer file of Qwen3.8-Flash-Next, because the model's own tokenizer.json has a different format; the counts were checked to be exact. (5) The 128K step ran with a prompt of 130,048 tokens (the rule of our speed test). Against the maker's own table. The maker's README lists, for the RTX 5090 D 24 GB, 2,169.9 tok/s prefill at 4K, 3,554.9 tok/s prefill at 16K, 99.7 tok/s decode in chat and 121.2 tok/s decode for an agent task, measured with the maker's own tool (bench_session.py, median of 3 runs). We ran the same tool of the maker with the maker's README start command (3 runs, median): 3,647.7 tok/s at 4K, 4,145.0 tok/s at 16K, 125.2 tok/s chat decode and 143.4 tok/s agent decode. These runs are not part of the published raw files of this page; the page's own numbers come from our speed test. The long-answer step wrote its reply to l1-output.txt. The context steps at 16K, 32K, 64K and 128K all fit the 131,072-token window.

python3 speed2.py --base-url http://127.0.0.1:46535/v1 --label edge0-145b-5090-1511-b-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/edge0-145b-5090-1511-b-speed-r0/edge0-145b-5090-1511-b-speed-r0/speed2 --contract $RUN/raw/edge0-145b-5090-1511-b-speed-r0/edge0-145b-5090-1511-b-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/s128/tok/qwen38-flashnext/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.0} --sampling-source The model card Edge0/Edge0-145B-q3k README.md @4136439d names no sampling. The author's own speed tool sends temperature 0 (edge0-strata/tools/strata_bench.py:133 @7b60076) and the author's docs say identical requests at temperature 0 can differ because decoding is not bit-for-bit repeatable (edge0-infer/docs/edge0-145b.md:131). Seed 42 sent anyway for speed. --model edge0_145b --no-cache-prompt --server-log $HOME/edge0-1511/logs/server.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 1 --server-slots-source docs: edge0-infer/docs/edge0-145b.md:129 (Edge0-AI/Edge0 @7b60076): 'Concurrent requests wait in edge0-strata's queue instead of receiving 429 as with the native server' = one request at a time; no batch flag in the author's launch (edge0-strata/edge0/models/edge0-145b.json, model-bench.json). --server-ctx 131072 --server-ctx-source edge0-infer/docs/edge0-145b.md:97 (--context default 131072) and edge0-strata/edge0/models/edge0-145b.json (--max-context 131072) @7b60076; HF README 'Context 256K' is the model maximum, not the author's launch value.

How well it answers

Thinking off (enable_thinking false in every request), temperature 0, seed 42. Raw numbers only, no overall score.

Thinking off in every request. Seed 42. Temperature 0 (the model card names no sampling; the maker's own speed tool sends 0). A cut-off answer counts as wrong. Measured 2026-11-15.

bash tools/quality2/measure_all.sh tools/quality2/cards/edge0-145b-q3k-edge0infer-7b60076-readme-5090-1511.json --parts mmlupro --job edge0-145b-5090-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/edge0-145b-q3k-edge0infer-7b60076-readme-5090-1511.json --parts gpqa --job edge0-145b-5090-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/edge0-145b-q3k-edge0infer-7b60076-readme-5090-1511.json --parts ifeval --job edge0-145b-5090-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/edge0-145b-q3k-edge0infer-7b60076-readme-5090-1511.json --parts lcb --job edge0-145b-5090-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/edge0-145b-q3k-edge0infer-7b60076-readme-5090-1511.json --parts ruler --job edge0-145b-5090-1511-q

bash tools/quality2/measure_all.sh tools/quality2/cards/edge0-145b-q3k-edge0infer-7b60076-readme-5090-1511.json --parts ruler --job edge0-145b-5090-1511-q

Also made this possible

• Model files (Edge0-145B, Q3_K, revision 4136439d)EEdge0 on Hugging FaceHugging Face
• Base model (Qwen3.8-Flash-Next)Qwen@Alibaba_Qwen Hugging Face GitHub Website
• Strata, which the engine builds onNNiko1221GitHub
• llama.cpp (ggml), part of the engine buildGggml-org@ggml_org GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one machine), driver 610.43.02, kernel 6.12.95 (Debian 13)

Model

Edge0-145B (Edge0, 145B latent sparse MoE, 10.12B active, base Qwen3.8-Flash-Next) Q3_K (HF Edge0/Edge0-145B-q3k @4136439d, pack 98.1 GiB incl. 47.7 GiB PLE table on SSD)

Model files (Edge0-145B, Q3_K, revision 4136439d)

huggingface.co/Edge0/Edge0-145B-q3k

Base model (Qwen3.8-Flash-Next)

huggingface.co/Qwen/Qwen3.8-Flash-Next

Engine

edge0-infer with edge0-strata at commit 7b60076, built from source — the program that loads the model and answers requests · github.com/Edge0-AI/Edge0

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   Engine start (the README command of the maker, with the default device and context written out and our own host and port): edge0-infer serve \
     --model Edge0-145B-q3k \
     --device cuda:0 \
     --host 127.0.0.1 \
     --port 8888 \
     --context 131072 (engine Edge0-AI/Edge0 at 7b60076, built from source with CUDA 13.3). The speed and quality tools of our tests then talk to it over the OpenAI-compatible API.

   What the parts mean

   • --model Edge0-145B-q3kthe model file it loads
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on

Edge0-AI/Edge0 edge0-infer serve + edge0-strata (fork of Niko1221/Strata), source build CUDA 13.3 sm_120, README standard start (template arguments only); Engine start (the README command of the maker, with the default device and context written out and our own host and port): edge0-infer serve --model Edge0-145B-q3k --device cuda:0 --host 127.0.0.1 --port 8888 --context 131072 (engine Edge0-AI/Edge0 at 7b60076, built from source with CUDA 13.3). The speed and quality tools of our tests then talk to it over the OpenAI-compatible API.