Home / Compare / GPQA with thinking on

Compare · GPQA Diamond · thinking on

GPQA Diamond with thinking on, one model per memory class

The same 198 GPQA Diamond questions, one try each, thinking on, on four setups: a 16 GB card, a 32 GB card, a 256 GB Mac and four DGX Sparks (512 GB). One model on each. Runs of 15 and 16 November 2026.

Results, biggest box first

Class and hardwareModel and quantEngine and versionCorrect of 198Share95 % range (Wilson)Cut off at the limit
512 GB
4× DGX Spark (GB10)
Inkling 2 Small
EXL3 4.0 bpw
0xSero/inkling2-exl3 @d982916317990.40 %85.50 to 93.7712
256 GB Mac
Mac Studio M5 Ultra 256 GB
Qwen3.8 Flash Next
MLX 6-bit
TensorFold 1.2.917588.38 %83.17 to 92.130
32 GB
RTX 5090 32 GB
Edge0-145B
Q3_K
Edge0-AI/Edge0 @7b60076641cc, engine 0.2.016683.84 %78.08 to 88.310
16 GB
RTX 5060 Ti 16 GB
Boole-32B-A2.5B Preview
Q2_0
llama.cpp fork by Precision Numerics @7b225ec310452.53 %45.59 to 59.3712
  • One try per question, seed 42, thinking on. An answer cut off at the token limit counts as wrong; the last column counts them. The 95 % range is the Wilson interval in percent. It covers the spread from the choice of questions, not from run to run (not measured).
  • All runs were on 16 November 2026, except Edge0-145B: it started on 15 November (22:46) and finished on 16 November (01:26).
  • Same for all four: the same 198 questions (list SHA-256 b0d857f94c14…), the same prompt template (checked on every row of every run), the same scorer, and thinking text in 198 of 198 answers of every run. This page shows counts only, no question or answer text.
  • These are setups, not equal-size models: each class has its own model, quant and engine, so the table shows what each size runs today.

What the numbers separate

The 16 GB run is clearly behind each of the other three: its 95 % range (45.59 to 59.37) does not overlap theirs, the paired test on the same questions gives p below 1e-13 for each of the three pairs (also after the Holm correction), and it stays behind even if all 12 of its cut-off answers were counted as right (116 of 198, 58.59 %). The ranges of the other three runs overlap in every pair (32 GB and 256 GB Mac, 32 GB and 512 GB, 256 GB Mac and 512 GB), so none of them counts as better than another.

One paired test points the other way, and we name it. On the same questions, Edge0-145B on the 32 GB card against Inkling 2 Small on four DGX Sparks: 5 questions only Edge0 got right, 18 only Inkling, p 0.0106 (exact McNemar), 0.0319 after the Holm correction over the six pairs. It does not count as “better” here: the 95 % ranges overlap, the Inkling run is stitched together (35 questions were drawn a second time, only the ones that had been cut off, which tends to lift its number somewhat; that is our reasoning, not a measurement), and 12 of its answers are still cut off at 32,768 tokens.

All six pairs, same 198 questions

PairCorrectOnly the first rightOnly the second rightp (exact McNemar)p (Holm)95 % ranges overlap
32 GB Edge0 against 512 GB Inkling166 / 1795180.01060.0319yes
256 GB Mac Flash Next against 512 GB Inkling175 / 17910140.5410.541yes
32 GB Edge0 against 256 GB Mac Flash Next166 / 17510190.1360.272yes
16 GB Boole against 512 GB Inkling104 / 1793787.3e-204.4e-19no
16 GB Boole against 256 GB Mac Flash Next104 / 1755762.3e-171.1e-16no
16 GB Boole against 32 GB Edge0104 / 1666682.1e-148.6e-14no

McNemar is two-sided, on the questions where the two runs differ. Holm is over the six pairs. A small p on a pair with overlapping ranges is shown here, not counted as “better”.

What differs between the runs

The questions, the template, the scorer, the seed and the thinking switch are the same. These settings are not, and each can move a number.

Setting512 GB, Inkling256 GB Mac, Flash Next32 GB, Edge016 GB, BooleWhat it can change
Token limit
thinking plus answer
16,384; 32,768 for 35 questions16,38416,38432,768 A limit only cuts. Edge0 and the Mac never reached it (longest answers 12,872 and 9,033 tokens). Boole has the most room: at 16,384, 76 of its answers would be cut and 82 of 198 would stay right (41.4 %), counted from the same run, not a new one. That helps Boole and changes no rank. Inkling can only gain (up to 191).
Thinking cap thinking on, no cap, no effort value sentthinking on, server option --thinking-budget 8192thinking on, no capthinking on, no cap The Mac cap was reached on 105 of 198 questions (85 right, 81 %); on the other 93, 90 were right (97 %). Without the cap the same setup got 134 of 198 in an earlier run on 15 November (61 cut off at 16,384, TensorFold 1.2.8). So the cap lifts the Mac number by up to 41 questions. The serve command of the Flash Next recipe (docs/recipes/qwen3.8-flash-next.md, line 19, TensorFold 1.2.9) has no --thinking-budget. The cap is our addition: without it, 61 of 198 answers ran into the 16,384 limit.
Sampling
seed 42 in all four
temperature 0, top_p 1.0temperature 1.0, top_p 0.95, top_k 20, min_p 0temperature 0temperature 1.0, top_p 0.95, top_k 20 A sampled run is one draw for its seed, and the spread between draws was not measured: direction unknown. At temperature 0, Edge0's documentation says runs do not repeat bit for bit, and all 35 Inkling re-draws differed from the first answers.
Requests at once 16211 The time per question, not the scoring. Through the batch mix, Inkling at temperature 0 can differ slightly: direction unknown, small.
Stitched runs 163 questions at 16,384 plus 35 re-draws at 32,768one run168 plus 30, same settings188 plus 10, same settings, two server starts Inkling's second draw is not a continuation of the first (all 35 differ), and only the cut-off questions got one, so 179 tends to sit somewhat above one full run at 32,768 (our reasoning, not measured). Boole's last 10 questions of the list ran apart: 8 right, none cut off, which is spread between questions, not a bias we can show. Edge0's two parts do not overlap.
Context window 380,000 per request262,144131,07265,536 Time and memory, not the answers: every window fits 16,384 or 32,768 tokens plus a short question. Boole runs at the author's evaluation context of 65,536 (its model card says 32,768); the Mac's window is forced below the author's 1,048,576.

Addition: correct among the answers that were not cut off

The table at the top counts a cut-off answer as wrong. This addition drops those questions and counts the rest. It depends on answer length (the model that thinks longest loses the most questions), so it is extra information and no basis for a ranking. The last column goes the other way: every cut-off answer counted as right, an upper bound that was not measured.

ClassQuestions not cut offCorrect among themShare95 % range (Wilson)If every cut-off answer were right
512 GB186179 of 18696.24 %92.44 to 98.17191 (96.46 %)
256 GB Mac198175 of 19888.38 %83.17 to 92.13same (nothing cut off)
32 GB198166 of 19883.84 %78.08 to 88.31same (nothing cut off)
16 GB186104 of 18655.91 %48.73 to 62.86116 (58.59 %)

Run pages of the four setups, with their speed, quality and memory tests. They show the standard tests with thinking off, and a shorter GPQA run with thinking on where one exists, not this 198-question run. Each page names its own engine build.

Credits. Inkling engine and quant: 0xSero (@0xSero). Inkling 2 Small: Thinking Machines (@thinkymachines). Flash Next recipe and engine (TensorFold): Ash (@ashxhart); base model Qwen (@Alibaba_Qwen). Edge0-145B, the quant and the engine: Edge0; base model Qwen. Boole, the quant and the llama.cpp fork: Precision Numerics (@precision_num); base model Flint by Recursive (@Recursive_SI).