Edge0-145B on one RTX 5090: the maker's engine, bench-file settings

How fast it runs

Long prompts

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; context steps: prompts of exactly 16K, 32K, 64K and 128K tokens (a filler log first, the code or prose task prompt last), 2 runs per step, 512 output tokens; decode = tokens after the first / time after the first token; prefill = prompt tokens / time to first token of the same request, median over its requests; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-11-15. Raw files: speed result · speed contract · log · long answer.

What ran. Edge0-145B (a latent sparse MoE with 145B parameters and 10.12B active, built on Qwen3.8-Flash-Next) in the Q3_K files of the maker (Hugging Face Edge0/Edge0-145B-q3k, revision 4136439d; 98.1 GiB on disk, of which 47.7 GiB are a table that stays on the SSD) on the maker's own engine: Edge0-AI/Edge0 at commit 7b60076 (edge0-infer serve with edge0-strata). The engine is built from source with CUDA 13.3 for the RTX 5090, as the maker describes for Linux; there is no Linux package. The context is 131,072 tokens, the maker's default, with one request at a time (further requests wait in the engine's queue). Where this run differs from the maker's setup. (1) Card: the maker's published numbers come from an RTX 5090 D with 24 GB; this run used an RTX 5090 with 32 GB. Nothing was lowered. (2) Build: only the target edge0-infer was built, because the build of the other targets stopped on a compiler test (the maker's pull request 221 is about it). (3) The file latent_dense_q4.bin was not loaded; it is only needed for the du=q4 setting. (4) Prompt lengths in the speed test are counted with the tokenizer file of Qwen3.8-Flash-Next, because the model's own tokenizer.json has a different format; the counts were checked to be exact. (5) The 128K step ran with a prompt of 130,048 tokens (the rule of our speed test). This page is the same engine as the README-start page with the flags of the maker's bench file. Against the maker's own table. The maker's README lists, for the RTX 5090 D 24 GB, 2,169.9 tok/s prefill at 4K, 3,554.9 tok/s prefill at 16K, 99.7 tok/s decode in chat and 121.2 tok/s decode for an agent task, measured with the maker's own tool (bench_session.py, median of 3 runs). We ran the same tool of the maker with the settings of the maker's bench file (model-bench.json), 3 runs, median: 2,838.1 tok/s at 4K, 2,897.4 tok/s at 16K, 100.2 tok/s chat decode and 131.3 tok/s agent decode. The 16K value is lower than with the README start command (4,145.0). The cause we found: the bench file's flag --no-prefill-borrow makes the engine cut the prefill into chunks of 2,048 tokens instead of 8,192 (edge0-strata generate.cpp lines 3835 to 3840 at 7b60076); the run reports a prefill cap of 2,048. Without that one flag and otherwise identical, the maker's tool measured 3,463.5 tok/s at 4K and 4,138.3 tok/s at 16K. We did not find with which flags the maker measured his table row. These runs are not part of the published raw files of this page.

python3 speed2.py --base-url http://127.0.0.1:39567/v1 --label edge0-145b-5090-1511-c-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/edge0-145b-5090-1511-c-speed-r0/edge0-145b-5090-1511-c-speed-r0/speed2 --contract $RUN/raw/edge0-145b-5090-1511-c-speed-r0/edge0-145b-5090-1511-c-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1,ctx --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --ctx-depths 16384,32768,65536,131072 --ctx-runs 2 --ctx-max-tokens 512 --tokenizer-json $HOME/s128/tok/qwen38-flashnext/tokenizer.json --tokenizer-python $HOME/not published --sampling {"temperature": 0.0} --sampling-source The model card Edge0/Edge0-145B-q3k README.md @4136439d names no sampling. The author's own speed tool sends temperature 0 (edge0-strata/tools/strata_bench.py:133 @7b60076) and the author's docs say identical requests at temperature 0 can differ because decoding is not bit-for-bit repeatable (edge0-infer/docs/edge0-145b.md:131). Seed 42 sent anyway for speed. --model edge0_145b --no-cache-prompt --server-log $HOME/edge0-1511/logs/server.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 1 --server-slots-source docs: edge0-infer/docs/edge0-145b.md:129 (Edge0-AI/Edge0 @7b60076): 'Concurrent requests wait in edge0-strata's queue instead of receiving 429 as with the native server' = one request at a time; no batch flag in the author's launch (edge0-strata/edge0/models/edge0-145b.json, model-bench.json). --server-ctx 131072 --server-ctx-source edge0-infer/docs/edge0-145b.md:97 (--context default 131072) and edge0-strata/edge0/models/edge0-145b.json (--max-context 131072) @7b60076; HF README 'Context 256K' is the model maximum, not the author's launch value.

Quality

Not run with these settings: no Quality v2 test, so the page shows no quality number. The README-start page of this model has the quality numbers.

Also made this possible

• Model files (Edge0-145B, Q3_K, revision 4136439d)EEdge0 on Hugging FaceHugging Face
• Base model (Qwen3.8-Flash-Next)Qwen@Alibaba_Qwen Hugging Face GitHub Website
• Strata, which the engine builds onNNiko1221GitHub
• llama.cpp (ggml), part of the engine buildGggml-org@ggml_org GitHub

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB (one machine), driver 610.43.02, kernel 6.12.95 (Debian 13)

Model

Edge0-145B (Edge0, 145B latent sparse MoE, 10.12B active, base Qwen3.8-Flash-Next) Q3_K (HF Edge0/Edge0-145B-q3k @4136439d, pack 98.1 GiB incl. 47.7 GiB PLE table on SSD)

Model files (Edge0-145B, Q3_K, revision 4136439d)

huggingface.co/Edge0/Edge0-145B-q3k

Base model (Qwen3.8-Flash-Next)

huggingface.co/Qwen/Qwen3.8-Flash-Next

Engine

edge0-infer with edge0-strata at commit 7b60076, built from source, with the flags of the maker's bench file — the program that loads the model and answers requests · github.com/Edge0-AI/Edge0

Steps

1. Get the engine and the model files from the list above.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8888, only from this computer.

   Copy

   Engine start (the README start command of the maker, with the default device and context written out and our own host and port, plus the maker's bench-file flags): edge0-infer serve --model Edge0-145B-q3k --device cuda:0 --host 127.0.0.1 --port 8888 --context 131072 plus --pool-workers 24 --pcie-mode dma --pcie-frac 0.1 --vram-reserve-mib 700 --kv-resident 0 --resident-experts --no-prefill-borrow --conversation-cache-mib 0 (from the maker's model-bench.json; engine Edge0-AI/Edge0 at 7b60076, built from source with CUDA 13.3).

   What the parts mean

   • --model Edge0-145B-q3kthe model file it loads
   • --host 127.0.0.1who can connect: only this computer
   • --port 8888the network port the server listens on

Edge0-AI/Edge0 edge0-infer serve + edge0-strata (fork of Niko1221/Strata), source build CUDA 13.3 sm_120, --resident-experts bench configuration of the author (model-bench.json); Engine start (the README start command of the maker, with the default device and context written out and our own host and port, plus the maker's bench-file flags): edge0-infer serve --model Edge0-145B-q3k --device cuda:0 --host 127.0.0.1 --port 8888 --context 131072 plus --pool-workers 24 --pcie-mode dma --pcie-frac 0.1 --vram-reserve-mib 700 --kv-resident 0 --resident-experts --no-prefill-borrow --conversation-cache-mib 0 (from the maker's model-bench.json; engine Edge0-AI/Edge0 at 7b60076, built from source with CUDA 13.3).