Home / Compare / GPQA with thinking on
GPQA Diamond with thinking on, one model per memory class
The same 198 GPQA Diamond questions, one try each, thinking on, on four setups: a 16 GB card, a 32 GB card, a 256 GB Mac and four DGX Sparks (512 GB). One model on each. Runs of 15 and 16 November 2026.
Results, biggest box first
| Class and hardware | Model and quant | Engine and version | Correct of 198 | Share | 95 % range (Wilson) | Cut off at the limit |
|---|---|---|---|---|---|---|
| 512 GB 4× DGX Spark (GB10) | Inkling 2 Small EXL3 4.0 bpw | 0xSero/inkling2-exl3 @d9829163 | 179 | 90.40 % | 85.50 to 93.77 | 12 |
| 256 GB Mac Mac Studio M5 Ultra 256 GB | Qwen3.8 Flash Next MLX 6-bit | TensorFold 1.2.9 | 175 | 88.38 % | 83.17 to 92.13 | 0 |
| 32 GB RTX 5090 32 GB | Edge0-145B Q3_K | Edge0-AI/Edge0 @7b60076641cc, engine 0.2.0 | 166 | 83.84 % | 78.08 to 88.31 | 0 |
| 16 GB RTX 5060 Ti 16 GB | Boole-32B-A2.5B Preview Q2_0 | llama.cpp fork by Precision Numerics @7b225ec3 | 104 | 52.53 % | 45.59 to 59.37 | 12 |
- One try per question, seed 42, thinking on. An answer cut off at the token limit counts as wrong; the last column counts them. The 95 % range is the Wilson interval in percent. It covers the spread from the choice of questions, not from run to run (not measured).
- All runs were on 16 November 2026, except Edge0-145B: it started on 15 November (22:46) and finished on 16 November (01:26).
- Same for all four: the same 198 questions (list SHA-256 b0d857f94c14…), the same prompt template (checked on every row of every run), the same scorer, and thinking text in 198 of 198 answers of every run. This page shows counts only, no question or answer text.
- These are setups, not equal-size models: each class has its own model, quant and engine, so the table shows what each size runs today.
What the numbers separate
The 16 GB run is clearly behind each of the other three: its 95 % range (45.59 to 59.37) does not overlap theirs, the paired test on the same questions gives p below 1e-13 for each of the three pairs (also after the Holm correction), and it stays behind even if all 12 of its cut-off answers were counted as right (116 of 198, 58.59 %). The ranges of the other three runs overlap in every pair (32 GB and 256 GB Mac, 32 GB and 512 GB, 256 GB Mac and 512 GB), so none of them counts as better than another.
One paired test points the other way, and we name it. On the same questions, Edge0-145B on the 32 GB card against Inkling 2 Small on four DGX Sparks: 5 questions only Edge0 got right, 18 only Inkling, p 0.0106 (exact McNemar), 0.0319 after the Holm correction over the six pairs. It does not count as “better” here: the 95 % ranges overlap, the Inkling run is stitched together (35 questions were drawn a second time, only the ones that had been cut off, which tends to lift its number somewhat; that is our reasoning, not a measurement), and 12 of its answers are still cut off at 32,768 tokens.
All six pairs, same 198 questions
| Pair | Correct | Only the first right | Only the second right | p (exact McNemar) | p (Holm) | 95 % ranges overlap |
|---|---|---|---|---|---|---|
| 32 GB Edge0 against 512 GB Inkling | 166 / 179 | 5 | 18 | 0.0106 | 0.0319 | yes |
| 256 GB Mac Flash Next against 512 GB Inkling | 175 / 179 | 10 | 14 | 0.541 | 0.541 | yes |
| 32 GB Edge0 against 256 GB Mac Flash Next | 166 / 175 | 10 | 19 | 0.136 | 0.272 | yes |
| 16 GB Boole against 512 GB Inkling | 104 / 179 | 3 | 78 | 7.3e-20 | 4.4e-19 | no |
| 16 GB Boole against 256 GB Mac Flash Next | 104 / 175 | 5 | 76 | 2.3e-17 | 1.1e-16 | no |
| 16 GB Boole against 32 GB Edge0 | 104 / 166 | 6 | 68 | 2.1e-14 | 8.6e-14 | no |
McNemar is two-sided, on the questions where the two runs differ. Holm is over the six pairs. A small p on a pair with overlapping ranges is shown here, not counted as “better”.
What differs between the runs
The questions, the template, the scorer, the seed and the thinking switch are the same. These settings are not, and each can move a number.
| Setting | 512 GB, Inkling | 256 GB Mac, Flash Next | 32 GB, Edge0 | 16 GB, Boole | What it can change |
|---|---|---|---|---|---|
| Token limit thinking plus answer |
16,384; 32,768 for 35 questions | 16,384 | 16,384 | 32,768 | A limit only cuts. Edge0 and the Mac never reached it (longest answers 12,872 and 9,033 tokens). Boole has the most room: at 16,384, 76 of its answers would be cut and 82 of 198 would stay right (41.4 %), counted from the same run, not a new one. That helps Boole and changes no rank. Inkling can only gain (up to 191). |
| Thinking cap | thinking on, no cap, no effort value sent | thinking on, server option --thinking-budget 8192 | thinking on, no cap | thinking on, no cap | The Mac cap was reached on 105 of 198 questions (85 right, 81 %); on the other 93, 90 were right (97 %). Without the cap the same setup got 134 of 198 in an earlier run on 15 November (61 cut off at 16,384, TensorFold 1.2.8). So the cap lifts the Mac number by up to 41 questions. The serve command of the Flash Next recipe (docs/recipes/qwen3.8-flash-next.md, line 19, TensorFold 1.2.9) has no --thinking-budget. The cap is our addition: without it, 61 of 198 answers ran into the 16,384 limit. |
| Sampling seed 42 in all four |
temperature 0, top_p 1.0 | temperature 1.0, top_p 0.95, top_k 20, min_p 0 | temperature 0 | temperature 1.0, top_p 0.95, top_k 20 | A sampled run is one draw for its seed, and the spread between draws was not measured: direction unknown. At temperature 0, Edge0's documentation says runs do not repeat bit for bit, and all 35 Inkling re-draws differed from the first answers. |
| Requests at once | 16 | 2 | 1 | 1 | The time per question, not the scoring. Through the batch mix, Inkling at temperature 0 can differ slightly: direction unknown, small. |
| Stitched runs | 163 questions at 16,384 plus 35 re-draws at 32,768 | one run | 168 plus 30, same settings | 188 plus 10, same settings, two server starts | Inkling's second draw is not a continuation of the first (all 35 differ), and only the cut-off questions got one, so 179 tends to sit somewhat above one full run at 32,768 (our reasoning, not measured). Boole's last 10 questions of the list ran apart: 8 right, none cut off, which is spread between questions, not a bias we can show. Edge0's two parts do not overlap. |
| Context window | 380,000 per request | 262,144 | 131,072 | 65,536 | Time and memory, not the answers: every window fits 16,384 or 32,768 tokens plus a short question. Boole runs at the author's evaluation context of 65,536 (its model card says 32,768); the Mac's window is forced below the author's 1,048,576. |
Addition: correct among the answers that were not cut off
The table at the top counts a cut-off answer as wrong. This addition drops those questions and counts the rest. It depends on answer length (the model that thinks longest loses the most questions), so it is extra information and no basis for a ranking. The last column goes the other way: every cut-off answer counted as right, an upper bound that was not measured.
| Class | Questions not cut off | Correct among them | Share | 95 % range (Wilson) | If every cut-off answer were right |
|---|---|---|---|---|---|
| 512 GB | 186 | 179 of 186 | 96.24 % | 92.44 to 98.17 | 191 (96.46 %) |
| 256 GB Mac | 198 | 175 of 198 | 88.38 % | 83.17 to 92.13 | same (nothing cut off) |
| 32 GB | 198 | 166 of 198 | 83.84 % | 78.08 to 88.31 | same (nothing cut off) |
| 16 GB | 186 | 104 of 186 | 55.91 % | 48.73 to 62.86 | 116 (58.59 %) |
Run pages and credits
Run pages of the four setups, with their speed, quality and memory tests. They show the standard tests with thinking off, and a shorter GPQA run with thinking on where one exists, not this 198-question run. Each page names its own engine build.
- 512 GB: Inkling 2 Small on four DGX Sparks. The page covers engine build 5cf1465d; this run used d9829163.
- 256 GB Mac: Qwen3.8 Flash Next 6-bit on a Mac Studio M5 Ultra. The page covers TensorFold 1.2.8; this run used 1.2.9.
- 32 GB: Edge0-145B on one RTX 5090, README start. Same engine commit as this run.
- 16 GB: Boole-32B-A2.5B Preview on one RTX 5060 Ti. Same engine commit; the page runs with a 32,768 window, this run with 65,536.
- How every test is run: How I test.
Credits. Inkling engine and quant: 0xSero (@0xSero). Inkling 2 Small: Thinking Machines (@thinkymachines). Flash Next recipe and engine (TensorFold): Ash (@ashxhart); base model Qwen (@Alibaba_Qwen). Edge0-145B, the quant and the engine: Edge0; base model Qwen. Boole, the quant and the llama.cpp fork: Precision Numerics (@precision_num); base model Flint by Recursive (@Recursive_SI).