niklaslenzKaitchup's Qwen3.8-27B-SLQ-EAR98 on vLLM, two RTX 5060 Ti
2x NVIDIA GeForce RTX 5060 Ti 16 GB (two cards of one machine, tensor parallel 2, PCIe, no NVLink).
Overall score as of 25 Oct 2026. Scores are relative to the best run in the class today; a better model later moves older runs down.
How the overall score is calculated
| Part | This run | Best in class | Relative | Weight | Points |
|---|---|---|---|---|---|
| Speed, one request | decode 33.0 tok/s | best 253.4 tok/s | 13.0 % | 17.5 % | 2.3 |
| Speed, 8 at once | total 143.1 tok/s | best 711.9 tok/s | 20.1 % | 17.5 % | 3.5 |
| Quality | mean of 7 tests: 65.6 % | best mean 68.0 % | 96.4 % | 25 % | 24.1 |
| Browser check | 8 of 10 | 80.0 % | 5 % | 4.0 | |
| My rating | 3 of 10 | 30.0 % | 35 % | 10.5 | |
| Overall | 100 % | 44.4 |
The seven quality tests: MMLU-Pro 58.8 · GPQA 42.4 · IFEval 83.0 · LiveCodeBench 55.0 · long text 8K 82.5 · long text 32K 67.5 · GPQA 30 with thinking 70.0.
The speed part is half the decode speed with one request and half the total speed with 8 requests at once, each relative to the best run of this class; a run without a total for 8 requests at once gets the better of its decode speed alone and the mix with its highest measured total. Quality is relative to the best run of this class, the browser check and my rating are out of 10. The parts are rounded one by one. sparkDash, the memory test and speculative-decoding numbers are not part of it. How the score is made, and the assumption behind it.
What it builds as an agent
Task: Airship dock (3D scene, one HTML file). One attempt, no help. Counted: the final file counts.
The exact promptcopy it and try it yourself↓
This is the task prompt word for word, as we handed it to the agent harness. The harness adds its own system prompt and tools, and some harnesses add short context notes of their own. Qwen Code 0.25.2 also put a space before six of the seven “@” signs in the library addresses of the import map (for example “three @0.170.0”), so the model got the prompt in that form. A run that goes past 90 minutes also gets a short “finish now” message, as the prompt says. To try it yourself, give it to your own coding agent: it needs a shell and a working folder.
Design and build the most impressive, creative, elaborate and detailed steampunk airship dock voxel world you are capable of. Theme: a steampunk airship dock. Everything else (style, story, extra details) is your choice.
How you work:
- You work alone in the folder /work. Nobody will answer questions. Aim to finish in about 60 minutes. After 90 minutes you will get a message to finish; then you have at most 30 more minutes. Hard stop after 120 minutes. Spend the time where it shows in the result.
- Tools on this machine: shell, files, python3 with numpy and Pillow, ffmpeg, node, and `shot.sh index.html` (opens the page in a headless browser, saves a screenshot and prints console errors). There is no internet and no access to other machines.
- You may create your own assets locally with these tools (textures, sky, sounds, music) and embed them in the file as data URIs. Do not download anything.
- Keep a working /work/index.html at all times: write a first working version early, then improve it step by step. If you are stopped, the last version that loaded without errors counts.
- Check your work with `shot.sh` as often as you like, fix every error, and keep improving until you are satisfied. Then stop.
Deliverable: ONE self-contained file /work/index.html that opens and runs in Chrome.
- JavaScript libraries: only these, through an import map with these exact URLs (they work offline in the test browser). Use whatever helps, including all Three.js addons (post-processing, sky, water, noise and so on):
- "three": "https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.module.js"
- "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/"
- "simplex-noise": "https://cdn.jsdelivr.net/npm/simplex-noise@4.0.3/dist/esm/simplex-noise.js"
- "@tweenjs/tween.js": "https://cdn.jsdelivr.net/npm/@tweenjs/tween.js@25.0.0/dist/tween.esm.js"
- "cannon-es": "https://cdn.jsdelivr.net/npm/cannon-es@0.20.0/dist/cannon-es.js"
- No other external resources. Build every object from cubes (voxels); InstancedMesh is allowed.
- The page fills the whole window and resizes with it. All visible text is in English.
- After the scene is built, set `window.scene` to the Three.js scene and `window.ready = true`.
- An automatic slow cinematic camera tour starts on load. Mouse input switches to OrbitControls (drag to rotate, scroll to zoom).
- Sound, if any, starts only after the first click (browsers block autoplay).
- A small label in the top-left corner: "Steampunk Airship Dock".
Minimum content (go far beyond it):
1. A tall iron docking tower with platforms, stairs or ladders, and a mooring arm at the top.
2. At least two airships, each with a balloon envelope, a gondola and propellers.
3. Big brass gears on the tower or the buildings.
4. Pipes and at least three chimneys that emit rising steam particles.
5. At least one crane and a stack of cargo crates.
6. A ground area with a workshop building and a small rail track.
7. Propellers and gears rotate, steam rises and fades, and one airship slowly approaches the tower, docks, waits and leaves again, in a loop.
8. Warm sunset lighting with shadows.
Play it yourself
Open full screenAI-generated The page this setup built, running live in your browser: click into it and use the controls it shows itself (usually drag to look around, scroll to zoom). It needs WebGL and can be slow on a weak graphics card.
The browser check
- page loads without page error1 / 1
- window.ready === true within 30 s1 / 1
- no console errors (whole session)1 / 1
- frame is not blank1 / 1
- frame rate in software GL (rAF over 5 s)0 / 1
- required objects (voxel mass, >=3 steam emitters, >=2 airships, label)2 / 2
- motion: pixel diff 0.5, >=3 rotating nodes 0.5, large group translating 12 / 2
- OrbitControls: drag and wheel change the picture beyond the animation baseline0 / 1
Measured 2026-10-25. Raw files: check · run info · command · final choice · contract · sampling · gate stats · gate log · screenshot · clip.mp4 · web.mp4 · web-poster.jpg.
The run's own headless check browser, which renders the page the model built, used up to 1215 % CPU. The load monitor counted it as another process. Browser check 8.0 out of 10 (airship-dock task): the page loads without a page error, is ready after 0.01 s, has no console errors, is not blank, shows the label "Steampunk Airship Dock" in the top left, the checker finds 3 airships (by their structure, not by name) and steam from 3 sources, and the scene moves (28 rotating parts, the motion check gave both of its points). The 2 lost points are the frame rate (7.63 frames per second in software rendering without a GPU, 10 give half a point and 20 the full point) and the orbit-controls test (dragging and the mouse wheel changed the picture less than the check needs). The agent (Qwen Code 0.25.2) took 103.8 minutes, one try, no help. After 90 minutes the harness sends “Time is up. Finish now”; it reached the model with the first request after that, at 96.5 minutes, and the run ended by itself at 103.8 minutes. 25 requests, none blocked, none with an error (973,908 prompt tokens, 161,560 answer tokens). The context of 94,208 tokens is smaller than the model's own: Qwen Code shortened its history four times, and my test proxy cut the answer limit to the room left in the context seven times. Two of those cuts (requests 2 and 3, at minutes 28 and 29) left only 16 tokens, because the proxy's size estimate for the prompt (101,278 and 90,268 tokens) was higher than the real size (66,245 and 54,116 tokens); both answers ended after 16 tokens and the agent went on. The first version of the page came after 71.0 minutes and did not load ("The requested module 'simplex-noise' does not provide an export named 'createNoise3'"); the first version that loaded without errors was seen at 99.4 minutes, and the last one, at 102.4 minutes, is the one that counts. The limit per answer was 262,144 tokens (the Qwen3.8-27B card, README lines 507 to 510). Thinking was on with that card's sampling (temperature 1.0, top_p 0.95, top_k 20). The load monitor saw other processes during the run (up to 1,215 % CPU, the run's own headless check browser).
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
run_showcase.sh --harness qwen-code --task K1 --upstream http://127.0.0.1:33685/v1 --model kaitchup/Qwen3.8-27B-SLQ-EAR98 --sampling-file $RUN/sampling-show.json --out $RUN/raw/nacht2410-show-r7/show --limit-s 7200 --inject-after-s 5400 --poll-s 20 --ctx 94208How fast it runs
| Measurement | Result |
|---|---|
| Decode | 33.0 tok/s |
| Words per second | 22.5 words/s |
| Characters per second | 157.3 chars/s |
| Decode, code prompt | 33.0 tok/s |
| Decode, prose prompt | 33.0 tok/s |
| Spread of the decode runs (CV) | 0.0 % |
| Spread of the code runs (CV) | 0.0 % |
| Spread of the prose runs (CV) | 0.0 % |
| Decode, window 1 | 33.0 tok/s |
| Prefill at 2K | 1024.3 tok/s |
| Prefill at 8K | 1013.1 tok/s |
| Prefill at 32K | 952.2 tok/s |
| Time to first token at 2K | 2.0 s |
| Time to first token at 8K | 8.1 s |
| Time to first token at 32K | 34.4 s |
| 1 request at once (total) | 30.5 tok/s |
| 2 at once (total) | 55.6 tok/s |
| 4 at once (total) | 91.9 tok/s |
| 8 at once (total) | 143.1 tok/s |
| 1 request at once (per request) | 30.5 tok/s |
| 2 at once (per request) | 27.8 tok/s |
| 4 at once (per request) | 23.0 tok/s |
| 8 at once (per request) | 18.0 tok/s |
| Server slots (from the recipe) | 1 slots |
| Long answer, wall time | 210.1 s |
Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-10-25. Raw files: speed result · speed contract · log · long answer.
Requests at once: the recipe's slot count is 1 (at the full context the KV cache holds about 1.01 requests of full length), but the server ran with --max-num-seqs 8 (my addition), so C1, C2, C4 and C8 are real parallel runs of short requests; long prompts at the same time would have to queue. The speed test ran the author's serve line with additions (kaitchup/Qwen3.8-27B-SLQ-EAR98 at revision 1165bcbb96ef, model card README lines 74 to 84): vLLM 0.31.0 with humming-kernels 0.1.16, --quantization humming --dtype bfloat16, the reasoning and tool-call options of the card (--reasoning-parser qwen3, --tool-call-parser qwen3_coder), no speculation (the author did not test it). Added by me, because the author tested on one RTX PRO 6000 Blackwell with 96 GB and the 23.7 GB weights do not fit one 16 GB card: --tensor-parallel-size 2 over both RTX 5060 Ti, --max-model-len 94208 instead of the author's 262,144 (262,144 failed with an out-of-memory error while vLLM captured its CUDA graphs; with --max-num-seqs 8 vLLM reported 94,864 as the largest length that fits after the weights, I ran 94,208), --max-num-seqs 8, and the environment variable VLLM_USE_FLASHINFER_SAMPLER=0 (the FlashInfer sampler did not compile here: the CUDA compiler and its headers do not match). The two cards talk over PCIe, there is no NVLink. Sampling for the speed test is the non-thinking sampling of the Qwen3.8-27B card (temperature 0.7, top_p 0.8, top_k 20, presence penalty 1.5), because the Kaitchup card gives none; seed 42; the requests ask for no prompt cache. vLLM's own prefix cache was on (its default). The prefill test sends a new prompt of each size three times and all three runs are equally fast (2K: 1024.3, 1024.2 and 1026.4 tok/s), so no cache hit is visible; vLLM does not report cached tokens, so I cannot prove it. The headline 33.0 tok/s is the mean of the code prompt (33.0) and the prose prompt (32.99), medians of two runs each (spread 0.017 % and 0.021 %). Requests at once: vLLM batches requests as they come, up to 8 here (--max-num-seqs 8, my addition). At the full 94,208-token context the KV cache holds 1.01 requests (vLLM log: "Maximum concurrency for 94,208 tokens per request: 1.01x"), which is the slot number of 1 in the register. The 2, 4 and 8 at once values were measured with the prose prompt and 512 tokens per request, not with full-context requests. Total speed grew from 30.5 tok/s with 1 request to 143.1 tok/s with 8, so the requests did run together, not one after the other.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 speed2.py --base-url http://127.0.0.1:41363/v1 --label nacht2410-speed-r5 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/nacht2410-speed-r5/nacht2410-speed-r5/speed2 --contract $RUN/raw/nacht2410-speed-r5/nacht2410-speed-r5/speed2/speed-contract.json --parts decode,prefill,conc,l1 --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --tokenizer-json $HOME/models-2410/kaitchup_Qwen3.8-27B-SLQ-EAR98/tokenizer.json --tokenizer-python $BENCH/venv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253 (not published) --model kaitchup/Qwen3.8-27B-SLQ-EAR98 --no-cache-prompt --server-log $RUN/logs/server-A.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 1 --server-slots-source vLLM 0.31.0 server log line 'Maximum concurrency for 94,208 tokens per request: 1.01x' ($HOME/queue-2410/logs/vllm-kaitchup-server.log) = KV-limited concurrency at the served context, floor, capped at 8; the author's line sets no --max-num-seqs (vLLM default applies, not a slot recipe)How well it answers
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| MMLU-Pro 400 fixed questions, letter only | 58.8 % ± 4.8 pts | 235 / 400 | 0 | answers · summary · log · contract | command |
| GPQA Diamond 198 questions, letter only | 42.4 % ± 6.8 pts | 84 / 198 | 0 | answers · summary · log · contract | command |
| IFEval 200 fixed prompts, strict | 83.0 % ± 5.2 pts | 166 / 200 | 0 | answers · summary · log · contract | command |
| LiveCodeBench 40 newest tasks, the code runs | 55.0 % ± 14.7 pts | 22 / 40 | 1 | summary · log · contract | command |
| Long text 8K 4 tasks x 10 at 8192 tokens | 82.5 % ± 11.6 pts | 33 / 40 | 0 | log · contract · answers · summary | command |
| Long text 32K 4 tasks x 10 at 32768 tokens | 67.5 % ± 14.0 pts | 27 / 40 | 0 | log · contract · answers · summary | command |
| GPQA 30, thinking on 30 questions, limit 16,384 tokens | 70.0 % ± 15.6 pts median 5,365 tokens, 163.4 s per question | 21 / 30 | 8 | contract · summary | command |
Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-10-25.
Note from the run: Quality ran with one request at a time (the register's slot count is 1), which is why it took about 3 hours. MMLU-Pro and GPQA Diamond ran with a short answer limit (48 tokens; the answer is one letter), IFEval with a longer one, LiveCodeBench with 32,768 tokens. Answers cut off at their limit count as wrong: 1 of the 40 LiveCodeBench answers (and 2 more had no code in them), 8 of the 30 GPQA answers with thinking (the limit of 16,384 tokens); no other test lost an answer. One answer file is not published: gpqa_think.jsonl (the thinking run). The filter for public files drops a file when text in it looks like a banned word (here longitude values with a trailing letter, which look like a unit figure). Its summary file is online. GPQA Diamond ran from the public mirror fingertap/GPQA-Diamond (198 questions), because the original is gated and I had no access token. Sampling: instruct tests with the Qwen3.8-27B card's non-thinking values (temperature 0.7, top_p 0.8, top_k 20, presence penalty 1.5, because the Kaitchup card names none), GPQA with thinking with its thinking values (temperature 1.0, top_p 0.95, top_k 20). These are my own tests, and no other run of this setup is mixed into the quality numbers on this page.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/kaitchup-qwen38-27b-slq-ear98-vllm0310-tp2-2x5060ti.json --parts mmlupro --job nacht2410bash tools/quality2/measure_all.sh tools/quality2/cards/kaitchup-qwen38-27b-slq-ear98-vllm0310-tp2-2x5060ti.json --parts gpqa --job nacht2410bash tools/quality2/measure_all.sh tools/quality2/cards/kaitchup-qwen38-27b-slq-ear98-vllm0310-tp2-2x5060ti.json --parts ifeval --job nacht2410bash tools/quality2/measure_all.sh tools/quality2/cards/kaitchup-qwen38-27b-slq-ear98-vllm0310-tp2-2x5060ti.json --parts lcb --job nacht2410bash tools/quality2/measure_all.sh tools/quality2/cards/kaitchup-qwen38-27b-slq-ear98-vllm0310-tp2-2x5060ti.json --parts ruler --job nacht2410bash tools/quality2/measure_all.sh tools/quality2/cards/kaitchup-qwen38-27b-slq-ear98-vllm0310-tp2-2x5060ti.json --parts ruler --job nacht2410bash tools/quality2/measure_all.sh tools/quality2/cards/kaitchup-qwen38-27b-slq-ear98-vllm0310-tp2-2x5060ti.json --parts think --job nacht2410How much context fits
| Context size | Prompttokens | Filled | First token | Prefilltok/s, as the server reports it | Decodetok/s | Peak memoryGiB | Result |
|---|---|---|---|---|---|---|---|
| 4K 4,096 tokens | 3,897 | 95.1 % | — | — | — | 29.21 | answered (stop) |
| 8K 8,192 tokens | 7,788 | 95.1 % | — | — | — | 29.22 | answered (stop) |
| 16K 16,384 tokens | 15,570 | 95.0 % | — | — | — | 29.22 | answered (stop) |
| 32K 32,768 tokens | 31,135 | 95.0 % | — | — | — | 29.22 | answered (stop) |
| 64K 65,536 tokens | 62,269 | 95.0 % | — | — | — | 29.22 | answered (length) |
| 92K 94,208 tokens | 89,514 | 95.0 % | — | — | — | 29.22 | answered (stop) |
Memory test. For each context size (4K, 8K, 16K ... up to the largest size tested for the setup) the server is started again with that context, one prompt that fills about 95 % of the context is sent, and the model must answer. The largest size that loaded and answered counts. Decode speed is measured on that answer at the largest size. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. Measured 2026-10-25. Raw files: result of every step · log · contract.
Note from the run: For each size from 4,096 to 94,208 tokens the server was started again with that size as context, one prompt that fills about 95 % of it was sent (89,514 tokens at the largest size), and the model answered at every size. 94,208 tokens is the largest context the two cards hold; the author's 262,144 does not fit (see the speed note). The memory number is the sum of the memory used on both cards, 29.22 GiB (about 14.6 GiB on each), and it is the same at every size, because vLLM reserves its KV cache when the server starts (GPU memory utilization 0.9, the default). This test measures no decode speed.
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
bash tools/quality2/measure_all.sh tools/quality2/cards/kaitchup-qwen38-27b-slq-ear98-vllm0310-tp2-2x5060ti.json --parts fit --job nacht2410Also made this possible
- Base model (Qwen3.8-27B)
QwenHugging Face @Alibaba_Qwen GitHub Website - Engine (vLLM)
vLLM (vllm-project)GitHub
Run it yourself
What you need and how to start it, step by step. The results below come from exactly this setup.
What you need
- Hardware
- 2x NVIDIA GeForce RTX 5060 Ti 16 GB (two cards of one machine, tensor parallel 2, PCIe, no NVLink).
- Model
- kaitchup/Qwen3.8-27B-SLQ-EAR98 (SLQ EAR>98, Humming-packed, 23.73 GB)
- Quantized model (SLQ EAR>98, Humming) and serve line
- huggingface.co/kaitchup/Qwen3.8-27B-SLQ-EAR98
- Base model (Qwen3.8-27B)
- huggingface.co/Qwen/Qwen3.8-27B
- Engine
- vLLM 0.31.0 with humming-kernels, tensor parallel 2 over two 16 GB cards, context 94,208 tokens — the program that loads the model and answers requests · github.com/vllm-project/vllmBuild and settings we used: vLLM 0.31.0 + humming-kernels 0.1.16 (pip wheels via mirror, uv, python 3.13), tensor-parallel-size 2, bf16 KV, no speculation
Steps
Get the engine and the model files from the list above.
Start the server with this command. It loads the model into memory.
vllm serve kaitchup/Qwen3.8-27B-SLQ-EAR98 \ --quantization humming \ --dtype bfloat16 \ --max-model-len 94208 \ --limit-mm-per-prompt '{"image":0,"video":0}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3
Exact command we measured withfor reproducing↓
You do not need this to run the model. It is the start text exactly as our run recorded it, with the engine build in front, so anyone can repeat the result.
vLLM 0.31.0 + humming-kernels 0.1.16 (pip wheels via mirror, uv, python 3.13), tensor-parallel-size 2, bf16 KV, no speculation; vllm serve kaitchup/Qwen3.8-27B-SLQ-EAR98 --quantization humming --dtype bfloat16 --max-model-len 94208 --limit-mm-per-prompt '{"image":0,"video":0}' --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3Context and sampling: 94,208 tokens (--max-model-len, the largest length that fits on two 16 GB cards after the weights; the author's line has 262,144, which does not fit and is not what runs here), bf16 KV cache (vLLM default), tensor parallel 2, --max-num-seqs 8, GPU memory utilization 0.9 (default), no speculation, seed 42. Sampling from the Qwen3.8-27B card, per part: speed, quality and sparkDash with thinking off use its non-thinking values (temperature 0.7, top_p 0.8, top_k 20, presence penalty 1.5; the Kaitchup card gives none); GPQA with thinking and the build test with thinking on use its thinking values (temperature 1.0, top_p 0.95, top_k 20, presence penalty 0). The answer limit for the build test was 262,144 tokens, the value the Qwen3.8-27B card names for reasoning (README lines 507 to 510; the Kaitchup card names none); my test proxy cuts it to the room left in the context for each request. Agent for the build test: Qwen Code 0.25.2, the agent of the Qwen family. Hardware: 2x NVIDIA GeForce RTX 5060 Ti 16 GB (two cards of one machine, tensor parallel 2, PCIe, no NVLink).. All raw files of this setup: raw/v2/qwen38-27b-kaitchup-slq-ear98-vllm-tp2-2x5060ti.
Mia's own benchmark, run unchanged
This is Mia's own benchmark tool (commit 21a3511), run unchanged. It has its own section on this page, and it is never mixed with my numbers. A cut-off answer counts as wrong here, so a value can differ from sparkDash's own screen.
| Test | Result95 % margin | Right | Cut offcount as wrong | Raw files | |
|---|---|---|---|---|---|
| Quality QA 150 questions | 77.3 % ± 6.7 pts | 116 / 150 | 0 | answers · summary | command |
| Quality MMLU 285 questions | 84.9 % ± 4.2 pts | 242 / 285 | 17 | answers · summary | command |
| Tool Eval Quick 15 scenarios, tool use | 87 / 100 | 12 / 15 passed | — | tool eval result · tool eval report · tool eval command · summary | command |
| Decode, 1 request at once first token 61 ms | 33.3 tok/s | — | — | decode result · summary | command |
| Prefill at 4K first token 4.1 s | 1047.1 tok/s | — | — | prefill result · summary | command |
| Prefill at 8K first token 8.2 s | 1019.1 tok/s | — | — | prefill result · summary | command |
| Prefill at 16K first token 16.6 s | 997.7 tok/s | — | — | prefill result · summary | command |
| Prefill at 32K first token 34.4 s | 955.7 tok/s | — | — | prefill result · summary | command |
Seed 1234 applies to the two quality tests; decode, prefill and tool eval send no seed. The benchmark client ran on a separate machine that is never part of the measured group. Measured 2026-10-25. Version 2.50.0, commit 21a3511.
QA. Quality QA, 150 items, seed 1234, temperature 0, thinking off, 4 requests at once (Mia's default). An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
MMLU. Quality MMLU, 285 items (5 for each of 57 subjects, 4 answers), seed 1234, temperature 0, thinking off. An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).
Decode. Decode bench, 1 request at once, 400 tokens, prompt type structured, temperature 0, thinking off. No seed is sent.
Prefill. Prefill bench at 4K, 8K, 16K and 32K tokens. sparkDash skips a size that does not fit into the context. No seed is sent.
Tool Eval Quick. Tool Eval Quick: tool-eval-bench (SeraphimSerapis, commit c61319b), 15 core scenarios (--short), --no-think, --timeout 120. Score 0 to 100. No seed is sent.
Note from the run: sparkDash 2.50.0 (@21a3511) on a client machine (never in the measured group), Quality: seed 1234, temperature 0, thinking off, a truncated answer counts as wrong; decode, prefill and tool eval send no seed; own row, never mixed with quality-v2. sparkDash runs its own tests (150 QA and 285 MMLU questions at temperature 0, thinking off, a cut-off answer counts as wrong; no QA answer and 17 of the 285 MMLU answers were cut off at the answer limit and count as wrong), so its decode and prefill values are not comparable with Speed v2 above: its decode test uses a structured prompt, Speed v2 a code and a prose prompt. The Tool Eval Quick test has 15 scenarios; 12 passed, which scores 87 of 100. The sparkDash client ran on another machine and was never part of the measured group.
Exact commands we measured withfor reproducing↓
You do not need this to run the model. It is the exact command of our measurement, so anyone can repeat the result.
python3 tools/quality2/sparkdash_client.py tools/quality2/cards/kaitchup-qwen38-27b-slq-ear98-vllm0310-tp2-2x5060ti.json <box> --client <client machine> --job nacht2410 --out $RUNtool-eval-bench --base-url http://127.0.0.1:<port> --model kaitchup/Qwen3.8-27B-SLQ-EAR98 --short --no-think --timeout 120 --json-file $BENCH/sparkdash-client/runs/nacht2410-sparkdash-r4/tooleval/result.json --no-live