Qwentin v0.14: Qwen3.8-27B (EXL3 5.75 bpw + MTP) on one RTX 5090

What it builds as an agent

Task: Airship dock (3D scene, one HTML file). One attempt, no help. Counted: the final file counts.

Design and build the most impressive, creative, elaborate and detailed steampunk airship dock voxel world you are capable of. Theme: a steampunk airship dock. Everything else (style, story, extra details) is your choice.

How you work:
- You work alone in the folder /work. Nobody will answer questions. Aim to finish in about 60 minutes. After 90 minutes you will get a message to finish; then you have at most 30 more minutes. Hard stop after 120 minutes. Spend the time where it shows in the result.
- Tools on this machine: shell, files, python3 with numpy and Pillow, ffmpeg, node, and `shot.sh index.html` (opens the page in a headless browser, saves a screenshot and prints console errors). There is no internet and no access to other machines.
- You may create your own assets locally with these tools (textures, sky, sounds, music) and embed them in the file as data URIs. Do not download anything.
- Keep a working /work/index.html at all times: write a first working version early, then improve it step by step. If you are stopped, the last version that loaded without errors counts.
- Check your work with `shot.sh` as often as you like, fix every error, and keep improving until you are satisfied. Then stop.

Deliverable: ONE self-contained file /work/index.html that opens and runs in Chrome.
- JavaScript libraries: only these, through an import map with these exact URLs (they work offline in the test browser). Use whatever helps, including all Three.js addons (post-processing, sky, water, noise and so on):
  - "three": "https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.module.js"
  - "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/"
  - "simplex-noise": "https://cdn.jsdelivr.net/npm/simplex-noise@4.0.3/dist/esm/simplex-noise.js"
  - "@tweenjs/tween.js": "https://cdn.jsdelivr.net/npm/@tweenjs/tween.js@25.0.0/dist/tween.esm.js"
  - "cannon-es": "https://cdn.jsdelivr.net/npm/cannon-es@0.20.0/dist/cannon-es.js"
- No other external resources. Build every object from cubes (voxels); InstancedMesh is allowed.
- The page fills the whole window and resizes with it. All visible text is in English.
- After the scene is built, set `window.scene` to the Three.js scene and `window.ready = true`.
- An automatic slow cinematic camera tour starts on load. Mouse input switches to OrbitControls (drag to rotate, scroll to zoom).
- Sound, if any, starts only after the first click (browsers block autoplay).
- A small label in the top-left corner: "Steampunk Airship Dock".

Minimum content (go far beyond it):
1. A tall iron docking tower with platforms, stairs or ladders, and a mooring arm at the top.
2. At least two airships, each with a balloon envelope, a gondola and propellers.
3. Big brass gears on the tower or the buildings.
4. Pipes and at least three chimneys that emit rising steam particles.
5. At least one crane and a stack of cargo crates.
6. A ground area with a workshop building and a small rail track.
7. Propellers and gears rotate, steam rises and fades, and one airship slowly approaches the tower, docks, waits and leaves again, in a loop.
8. Warm sunset lighting with shadows.

The browser check

• page loads without page error1 / 1
• window.ready === true within 30 s1 / 1
• no console errors (whole session)1 / 1
• frame is not blank1 / 1
• frame rate in software GL (rAF over 5 s)0 / 1
• required objects (voxel mass, >=3 steam emitters, >=2 airships, label)2 / 2
• motion: pixel diff 0.5, >=3 rotating nodes 0.5, large group translating 12 / 2
• OrbitControls: drag and wheel change the picture beyond the animation baseline0 / 1

Measured 2026-10-25. Raw files: check · run info · command · final choice · contract · sampling · screenshot · clip.mp4 · web.mp4 · web-poster.jpg.

The run's own headless check browser, which renders the page the model built, used up to 1515 % CPU. The load monitor counted it as another process. Browser check 8.0 out of 10 (airship-dock task): the page loads without a page error, is ready after 0.01 s, has no console errors, is not blank, shows the required objects (2 airships, 3 steam emitters, the label "Steampunk Airship Dock" top left) and moves (the motion check gave its 2 points). The 2 lost points are the frame rate (4.14 frames per second in software rendering without a GPU, 10 give half a point and 20 the full point) and the orbit-controls test (drag and wheel changed the picture less than the check needs). The agent (Qwen Code 0.25.2) finished by itself after 15.4 minutes, one try, no help; the "finish now" message (from 90 minutes) was not needed. It wrote the whole page in one step: the first request ran 8.3 minutes and ended with one write_file of 35,950 bytes, then it ran the page checker, fixed one problem with one edit (35,941 bytes, after about 13 minutes) and stopped. 2 versions of the page were seen on the way, both loaded without errors, and the last one counts. 15 requests, no error, no request blocked. The server was the author's, which decodes greedily (see the speed note). The last message of the model repeats the same few lines over and over (greedy decoding); it came after the page was saved and has no effect on the page. The number of tokens the model wrote cannot be given for this run: the author's server sends no usage figures when a request carries tools, and it drops the thinking text of a tool-call answer (serve_batched.py:1255, 1269-1290). The token field in my tool's raw files is a count of stream chunks and not a token count, so this page shows no token figure and the published copy of contract-show.json has that field emptied; the original on the box is unchanged. The limit per answer was 262,144 tokens, the value on the Qwen3.8-27B model card (README lines 507 to 510); Qwen Code itself sent 212,298 to 225,587 as the limit per request; no answer reached it. Thinking was on (through the shim); the sampling my tools send is the model card's (temperature 1.0, top_p 0.95, top_k 20) and is ignored by the server.

run_showcase.sh --harness qwen-code --task K1 --upstream http://127.0.0.1:33705/v1 --model qwen3.8-27b --sampling-file $RUN/sampling-show.json --out $RUN/raw/nacht2410-show-r7/show --limit-s 7200 --inject-after-s 5400 --poll-s 20 --ctx 262144

Extra build: Voxel world (not part of the score)

The 32 GB class is ranked on the airship-dock task. This voxel-world run used a different prompt (the one the 128 GB+ classes use), so it is shown as an extra and does not count toward the score or the ranking.

Task: Voxel world (3D scene, one HTML file). One attempt, no help. Which file is used: no version loaded, nothing counts.

Design and create the most impressive, creative, elaborate and detailed
voxel art world you are capable of. The theme and style are ENTIRELY your
choice — pick whatever world you can execute best and push it as far as
you can: scale, variety, detail, color, animation, atmosphere, life.
Surprise me.

Hard requirements:
- Everything ends up in ONE single self-contained HTML file (game.html in
  the current working directory) that opens and runs directly in Chrome.
  Use WebGL and whatever libraries you like, but embed everything —
  no internet access at view time, no external files.
- The scene must present itself: include an automatic slow cinematic
  camera tour (orbiting the world, occasionally zooming in on highlights)
  that starts on load, plus mouse controls (orbit/zoom) for manual viewing.
- All user-facing text (UI, HUD, titles, descriptions, hints) must be in
  ENGLISH, regardless of the language of any project files you encounter.

When finished, verify your HTML actually loads and renders without
console errors, then state DONE with the absolute path of game.html.
Do not ask questions.

How you work (same for every model):
- You work alone in the folder /work (this is the current working directory). Nobody will answer questions. Aim to finish in about 60 minutes. After 90 minutes you will get a message to finish; then you have at most 30 more minutes. Hard stop after 120 minutes.
- Tools on this machine: shell, files, python3 with numpy and Pillow, ffmpeg, node, and `shot.sh game.html` (opens the page in a headless browser, saves a screenshot and prints console errors). There is no internet and no access to other machines.
- Keep a working /work/game.html at all times. If you are stopped, the last version that loaded without errors counts.
- JavaScript libraries available offline in the test browser, through an import map with these exact URLs: "three": "https://cdn.jsdelivr.net/npm/three@0.170.0/build/three.module.js", "three/addons/": "https://cdn.jsdelivr.net/npm/three@0.170.0/examples/jsm/", "simplex-noise": "https://cdn.jsdelivr.net/npm/simplex-noise@4.0.3/dist/esm/simplex-noise.js", "@tweenjs/tween.js": "https://cdn.jsdelivr.net/npm/@tweenjs/tween.js@25.0.0/dist/tween.esm.js", "cannon-es": "https://cdn.jsdelivr.net/npm/cannon-es@0.20.0/dist/cannon-es.js". No other external resources.
- After the scene is built, set `window.scene` to the Three.js scene (if you use Three.js) and `window.ready = true`.

The browser check

• page loads without page error0 / 1
• window.ready === true within 30 s0 / 1
• no console errors (whole session)0 / 1
• frame is not blank0 / 1
• frame rate in software GL (rAF over 5 s)0 / 1
• world: scene size 0.5 (parts >= 80 or >= 20000 triangles; canvas fallback), visible text English if any 0.50 / 1
• automatic camera tour: the view changes on its own in two successive ~5 s windows (1 each)0 / 2
• animation in the world (nodes / instances / particles moving besides the camera)0 / 1
• OrbitControls: drag and wheel change the picture beyond the animation baseline0 / 1

Measured 2026-10-25. Raw files: check · run info · command · final choice · contract · sampling.

Extra build, not part of the score. This run used the voxel-world prompt (the one the 128 GB and larger classes use), not the airship-dock prompt this page is ranked on. Same box, same server, same agent as the airship-dock run above; only the prompt differs. It failed, twice the same: I ran it a second time with the server's token counting fixed (my request shim then adds the usage figures that the server omits for tool-call answers) and got the same result. In the first request the model thought for 125,941 tokens (606 s) and never reached a tool call to write the page: the answer ends in the middle of a sentence (finish reason stop, no end-of-thinking marker). Two short follow-up tool calls (writing a memory note and reading it back) came after that. The page game.html was never written, so there is nothing to play and the check scored 0 of 10 (index.html missing). None of my limits is the cause: the answer limit was 262,144 tokens, the context 262,144, and the largest prompt had 133,563 tokens. The server decodes greedily, so a second run gives the same result. A different setting from mine (sampling, thinking off, another agent) would be a different setup from the author's, so I did not try one. 3 requests, 131,460 answer tokens, 13.1 minutes.

run_showcase.sh --harness qwen-code --task VX --upstream http://127.0.0.1:43713/v1 --model qwen3.8-27b --sampling-file $RUN/sampling-show.json --out $RUN/raw/nacht2410-show-r0/show --limit-s 7200 --inject-after-s 5400 --poll-s 20 --ctx 262144

How fast it runs

Method: code+prose; 2 run(s) each; 512 tokens; 1 warm-up(s); median; requests at once: C1,C2,C4,C8 at once; 2 rounds each; 512 tokens per request; prose prompt; total = all tokens / wall time, per request = tokens / end-to-end time; median over the rounds. Measured 2026-10-25. Raw files: speed result · speed contract · log · long answer.

Requests at once: the server has 12 slots (from the author's recipe). The author's own serve line (kacper-daftcode/qwentin at revision 8a62637, README.md:86-88): serve_batched.py --port 8100 --max-slots 12 --num-blocks 4500 --timeout 1800 --model-name qwen3.8-27b --tqf ... --model-dir ... --lib build-qwen/libforward_qwen.so. Where my setup differs from the author's README: (1) I converted the weights myself, as the README describes (exllamav3 02aef45, the recipe qwen3_8-27b-exl3-opt5.75.yaml, README.md:77-83), from the original BF16 Qwen3.8-27B; (2) I built the engine with CUDA 13.1.115, the README's version, but my library file differs from the one the author lists (md5 8ce643946d90 against his 8f42ffdde443), so it is a different build of the same sources; (3) the serve line has one addition, --max-tokens-cap 262144, because the server's default answer cap is 16,384 tokens (tools/serve_common.py:8) and the build test needs the model card's larger limit; (4) a small request shim in front of the server (port 8888 to 8100) copies chat_template_kwargs.enable_thinking into the field enable_thinking, the only form the server reads (serve_batched.py:1239); without it every test marked thinking-on would have run without thinking. Nothing else is changed; answers pass through. The server always decodes greedily: it reads only max_tokens, stream, tools, enable_thinking and stream_options from a request (serve_batched.py:1223-1254) and the engine returns the arg-max token (:577); the author's own README example sends temperature 0.0 (README.md:89). So the sampling values and the seed that my tools send are ignored by this server, in every test on this page. Slots: the author's line sets 12 slots (README.md:87, with a pooled KV cache of 4,500 blocks = 576K tokens), so the concurrency test ran all four levels, 1, 2, 4 and 8 requests at once, inside the slots. Context 262,144 tokens (the engine's default, src/engine/config.cuh:28), 4-bit trellis KV cache, MTP speculative decoding on (default), the server's own cache of earlier prompts on (the author's default). The headline 148.74 tok/s is the mean of the code prompt (158.28) and the prose prompt (139.19), medians of two runs each (spread 7.1 % and 1.4 %). 500 words take about 4.9 s (500 / 101.69 words per second). The concurrency test sends the prose prompt: 121.45 tok/s for 1 request (this value counts the whole request including the wait for the first token, so it is below the decode value), 185.06, 262.38 and 332.22 tok/s in total for 2, 4 and 8 at once, which is 94.92, 70.15 and 46.81 tok/s per request. The engine reports no draft statistics, so the acceptance of the drafts is not available. The author's README lists higher decode values for 1 stream at 32K context (README.md:22), from his own test on his own box; my test uses fresh prompts of 512 tokens (code and prose) and is not the same test, so the two sets of numbers cannot be compared.

python3 speed2.py --base-url http://127.0.0.1:32967/v1 --label nacht2410-speed-r0 --prompts $BENCH/repos/rigmark-v2/prompts-v2.json --output $RUN/raw/nacht2410-speed-r0/nacht2410-speed-r0/speed2 --contract $RUN/raw/nacht2410-speed-r0/nacht2410-speed-r0/speed2/speed-contract.json --parts decode,prefill,conc,l1 --prefill-depths 2048,8192,32768 --prompt-ids code,prose --runs 2 --max-tokens 512 --warmups 1 --prefill-runs 3 --conc-levels 1,2,4,8 --conc-rounds 2 --conc-max-tokens 512 --conc-prompt prose --tokenizer-json $HOME/models-2410/Qwen_Qwen3.8-27B/tokenizer.json --tokenizer-python $HOME/hfvenv/bin/python --sampling {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5} --sampling-source https://huggingface.co/Qwen/Qwen3.8-27B README.md lines 252 (Thinking Mode) and 253 (Instruct/non-thinking mode), local copy not published line 252-253 (not published) --model qwen3.8-27b --no-cache-prompt --server-log $RUN/logs/server-A.log --engine-version-file $RUN/logs/engine-version.txt --server-slots 12 --server-slots-source kacper-daftcode/qwentin@8a62637 (v0.14, engine 49bc602) README.md:87 (--max-slots 12; --num-blocks 4500 = 576K pooled tokens)

How well it answers

Thinking off, except the last test. Seed 42. Sampling from the model card. A cut-off answer counts as wrong. Measured 2026-10-25.

Note from the run: The server always decodes greedily: it reads only max_tokens, stream, tools, enable_thinking and stream_options from a request (serve_batched.py:1223-1254) and the engine returns the arg-max token (:577); the author's own README example sends temperature 0.0 (README.md:89). So the sampling values and the seed that my tools send are ignored by this server, in every test on this page. The questions ran 12 at a time, the slot count of the author's line (same questions, limits and seed 42 for every question); the median time per question of the thinking test (61.1 s) was measured under that load. MMLU-Pro and GPQA Diamond ran with a short answer limit (the answer is one letter), none cut off. Answers cut off at their limit count as wrong: 2 of the 40 LiveCodeBench answers (limit 32,768 tokens; 1 more answer had no code) and 8 of the 30 GPQA answers with thinking (limit 16,384 tokens); no other test lost an answer. GPQA Diamond ran from the public mirror fingertap/GPQA-Diamond (198 questions), because the original is gated and I had no access token. Sampling as my tools send it: instruct tests the Qwen3.8-27B card's non-thinking values (temperature 0.7, top_p 0.8, top_k 20, presence penalty 1.5), GPQA with thinking the card's thinking values (temperature 1.0, top_p 0.95, top_k 20); the server ignores them (see above). These are my own tests, and no other run of this setup is mixed into the quality numbers on this page.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwentin-qwen38-27b-exl3-5.75bpw-v014-5090.json --parts mmlupro --job nacht2410

bash tools/quality2/measure_all.sh tools/quality2/cards/qwentin-qwen38-27b-exl3-5.75bpw-v014-5090.json --parts gpqa --job nacht2410

bash tools/quality2/measure_all.sh tools/quality2/cards/qwentin-qwen38-27b-exl3-5.75bpw-v014-5090.json --parts ifeval --job nacht2410

bash tools/quality2/measure_all.sh tools/quality2/cards/qwentin-qwen38-27b-exl3-5.75bpw-v014-5090.json --parts lcb --job nacht2410

bash tools/quality2/measure_all.sh tools/quality2/cards/qwentin-qwen38-27b-exl3-5.75bpw-v014-5090.json --parts ruler --job nacht2410

bash tools/quality2/measure_all.sh tools/quality2/cards/qwentin-qwen38-27b-exl3-5.75bpw-v014-5090.json --parts ruler --job nacht2410

bash tools/quality2/measure_all.sh tools/quality2/cards/qwentin-qwen38-27b-exl3-5.75bpw-v014-5090.json --parts think --job nacht2410

How much context fits

Memory test. The recipe fixes the context (262,144 tokens), so the server is started once with the recipe's own command and reserves its memory up front. Then, for each size (4K, 8K, 16K ... up to the largest size tested), one prompt that fills about 95 % of that size is sent, and the model must answer. The largest size that answered counts; it is the highest size tested, not a limit of the setup. Decode speed at that size was not measured. Peak memory is the GPU memory in use (nvidia-smi); the GPU was empty before each start. Measured 2026-10-25. Raw files: result of every step · log · contract.

Note from the run: fit memory per node, highest rise over idle: local 29.7 GiB The server loads once with its fixed context of 262,144 tokens (pooled KV cache, 4,500 blocks); the memory number does not grow with the prompt. For each size from 4K to 128K one prompt that fills about 95 % of the size was sent and the model answered at every size. The largest size tested is 131,072 tokens (a prompt of 124,545 tokens, 65.4 s); I did not send a prompt of 262,144 tokens. The memory number is the GPU memory in use on the 5090: 29.65 GiB at 4,096 tokens and 29.67 GiB at the largest size. The card has 32 GB. Decode speed at these sizes is not measured.

bash tools/quality2/measure_all.sh tools/quality2/cards/qwentin-qwen38-27b-exl3-5.75bpw-v014-5090.json --parts fit --job nacht2410

Also made this possible

• ModelQwenHugging Face @Alibaba_Qwen GitHub Website
• EXL3 quantizer (exllamav3)turboderp@turboderp_ GitHub Hugging Face

Run it yourself

What you need and how to start it, step by step. The results below come from exactly this setup.

What you need

Hardware

1x NVIDIA GeForce RTX 5090 32 GB, one GPU.

Model

Qwen3.8-27B EXL3 opt-5.75 bpw + MTP drafter (self-converted with exllamav3 02aef45 per the Qwentin README; Qwentin .tqf)

Engine

Qwentin engine (kacper-daftcode) for one RTX 5090, EXL3 5.75 bpw weights, 4-bit trellis KV cache, MTP drafts, 12 slots, 262,144 tokens of context — the program that loads the model and answers requestsBuild and settings we used: Qwentin v0.14 (engine 49bc602) serve_batched.py, 12 slots, MTP spec decoding, prefix cache

Steps

1. The list has no link for the engine (Qwentin engine (kacper-daftcode) for one RTX 5090, EXL3 5.75 bpw weights, 4-bit trellis KV cache, MTP drafts, 12 slots, 262,144 tokens of context). The list has no direct download link for the model file itself. The base model is huggingface.co/Qwen/Qwen3.8-27B.

2. Start the server with this command. It loads the model into memory. When the model has loaded, the server waits for requests on port 8100.

   Copy

   CUDA_VISIBLE_DEVICES=0 python3 \
     -u tools/serve_batched.py \
     --port 8100 \
     --max-slots 12 \
     --num-blocks 4500 \
     --timeout 1800 \
     --model-name qwen3.8-27b \
     --tqf qwen3_8-27b-exl3-opt5.75-mtp4.tqf \
     --model-dir Qwen3.8-27B \
     --lib build-qwen/libforward_qwen.so

   What the parts mean

   • CUDA_VISIBLE_DEVICES=0which NVIDIA GPU the program may use: the number of the GPU in our machine (with one GPU it is 0)
   • --model-dir Qwen3.8-27Bthe folder with the model files
   • --port 8100the network port the server listens on

Qwentin v0.14 (engine 49bc602) serve_batched.py, 12 slots, MTP spec decoding, prefix cache; CUDA_VISIBLE_DEVICES=0 python3 -u tools/serve_batched.py --port 8100 --max-slots 12 --num-blocks 4500 --timeout 1800 --model-name qwen3.8-27b --tqf qwen3_8-27b-exl3-opt5.75-mtp4.tqf --model-dir Qwen3.8-27B --lib build-qwen/libforward_qwen.so

Mia's own benchmark, run unchanged

This is Mia's own benchmark tool (commit 21a3511), run unchanged. It has its own section on this page, and it is never mixed with my numbers. A cut-off answer counts as wrong here, so a value can differ from sparkDash's own screen.

Seed 1234 applies to the two quality tests; decode, prefill and tool eval send no seed. The benchmark client ran on a separate machine that is never part of the measured group. Measured 2026-10-25. Version 2.50.0, commit 21a3511.

QA. Quality QA, 150 items, seed 1234, temperature 0, thinking off, 4 requests at once (Mia's default). An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).

MMLU. Quality MMLU, 285 items (5 for each of 57 subjects, 4 answers), seed 1234, temperature 0, thinking off. An answer cut off at the token limit counts as wrong. ± is the 95 % margin (Wilson).

Decode. Decode bench, 1 request at once, 400 tokens, prompt type structured, temperature 0, thinking off. No seed is sent.

Prefill. Prefill bench at 4K, 8K, 16K and 32K tokens. sparkDash skips a size that does not fit into the context. No seed is sent.

Tool Eval Quick. Tool Eval Quick: tool-eval-bench (SeraphimSerapis, commit c61319b), 15 core scenarios (--short), --no-think, --timeout 120. Score 0 to 100. No seed is sent.

Note from the run: sparkDash runs its own tests (its own QA and MMLU questions at temperature 0, thinking off, a cut-off answer counts as wrong; none of the 150 QA answers and 22 of the 285 MMLU answers were cut off at the answer limit and count as wrong), so its decode and prefill values are not comparable with Speed v2 above: its decode test uses a structured prompt (314.55 tok/s at one request), Speed v2 a code and a prose prompt. The sparkDash client ran on another machine and was never part of the measured group.

python3 tools/quality2/sparkdash_client.py tools/quality2/cards/qwentin-qwen38-27b-exl3-5.75bpw-v014-5090.json <box> --client <client machine> --job nacht2410 --out $RUN

tool-eval-bench --base-url http://127.0.0.1:<port> --model qwen3.8-27b --short --no-think --timeout 120 --json-file $BENCH/sparkdash-client/runs/nacht2410-sparkdash-r8/tooleval/result.json --no-live