Our number was right for the wrong reason
In September, Arbiter's answer accuracy went from 0.575 to 0.975 after a de-identification fix. That jump was real. But when I ran a power analysis on the harness, I found that with 40 gold questions it could not detect any difference smaller than about 15 points 7. The 0.575 to 0.975 move cleared that bar easily. A 3-point improvement, or a 3-point regression, would have been invisible.
How big a change an eval can see, by its size
Two-sided test at 5% significance with 80% power, accuracy variance 0.111
Source: Arbiter calculation, formula in Show your work 7
Show the data
| questions | min_detectable_points |
|---|---|
| 20 | 20.86 |
| 30 | 17.03 |
| 40 | 14.75 |
| 50 | 13.19 |
| 60 | 12.04 |
| 80 | 10.43 |
| 100 | 9.33 |
| 125 | 8.34 |
| 150 | 7.62 |
| 200 | 6.60 |
| 250 | 5.90 |
| 300 | 5.39 |
| 346 | 5.02 |
| 400 | 4.66 |
| 500 | 4.17 |
| 600 | 3.81 |
| 700 | 3.53 |
| 800 | 3.30 |
| 900 | 3.11 |
| 969 | 3.00 |
| 1000 | 2.95 |
The curve flattens fast. Going from 40 to 346 questions cuts the detectable change from 14.7 points to 5.0. Getting to 3 points takes 969. The generator already writes the questions, so moving from 40 to 346 cost nothing but run time. Card and colleagues made the same point about NLP test sets in 2020: many are too small to detect the gains papers claim 9.
The field mostly doesn't check
The largest review so far, of 445 LLM benchmark papers from the main ML and NLP venues, found that 16.0% used any uncertainty estimate or statistical test when comparing models 1. Tools are part of the reason. Of 57 open-source evaluation harnesses surveyed this year, fewer than a quarter support uncertainty estimates at all 5:
Share that report uncertainty at all
Three different populations, each counted by its own study
Source: Bean et al. 2025 1; Zhao et al. 2026 5; Reuel et al. 2024 6
Show the data
| label | value | display | note |
|---|---|---|---|
| LLM benchmark papers, 2018-24 (445) | 16.0 | 16% | used uncertainty estimates or statistical tests to compare results; Bean et al. 2025 |
| Open-source evaluation harnesses (57) | 22.8 | 23% | support uncertainty quantification at all; Zhao et al. 2026, preprint |
| AI benchmarks scored by BetterBench (24) | 41.7 | 42% | ran multiple evaluations or reported significance (10 of 24); Reuel et al. 2024 |
Medical benchmarks have their own version of this gap. Of 56 medical LLM benchmarks reviewed at ACL 2026, 88% do not address data contamination, 89% have no robustness test, and 91% never test how a model handles uncertainty 2. That last figure is about the model's behavior, such as saying "I don't know", not about error bars. It is the behavior a clinical tool most needs.
The harness is part of the result
A score depends on more than the model. A July 2026 preprint ran 12 open-weight models through 26 harness configurations on the same 3,679 items. One model's accuracy ranged from 30.6% to 88.6% 4:
One model, same questions, different harness settings
Accuracy (%) across 26 harness configurations
Source: Parupudi, 2026, single-author preprint 4
Show the data
| label | before | after | tag |
|---|---|---|---|
| gemma4-31b, same 3,679 items | 30.6 | 88.6 | 26 harness configurations |
Four of the twelve models reached first place under some configuration. The author notes that two of those wins were by 1 and 3 items, so treat the count as fragile. The direction is still clear. The UK AI Security Institute reports a similar effect for compute: on two coding benchmarks, raising the token budget from 1M to 10M raised scores by about 25%, while a health benchmark did not improve 8.
What Arbiter does about it
- 346 gold questions per run, in four kinds, each graded by kind.
- A release gate. The agent's policy refuses to call a release verified unless the eval has at least 346 questions (
F-EVAL-POWER), strict de-identification recall is at least 0.99, and there are zero cross-patient leaks. - The gate fails today, and that is the honest state. The last full eval on the running stack had 40 questions. The 346-question run needs the full Docker stack, and it is next.
Next: error bars on every reported accuracy, clustered by patient record, as Miller recommends for grouped questions 3.
Show your work
How this was done
The power curve is the standard sample-size formula for detecting a difference in accuracy: the smallest detectable difference is (z0.975 + z0.80) × √(σ² / n) = 2.80 × √(0.111 / n). The variance 0.111 is the value that reproduces Arbiter's published table (969 questions for 3 points, 349 for 5, 87 for 10, 39 for 15) 7. It corresponds to accuracy near 87%, or to paired comparisons, and the exact value depends on the accuracy level and the design. The code is min_detectable_points() in site/charts.py, and the CSV holds every point drawn.
The field figures are as each study reports them, read in the papers on 2 November 2026 1456. The UK AI Security Institute's token-budget result is described in the text, not charted, because its units are not stated 8.
What this does not tell you
The three bars count different things: papers, harness software and scored benchmarks. Read them side by side, not as one trend. Two of the cited studies are preprints, and the harness study has one author, who calls its "4 of 12 models reach first place" count fragile. The power formula assumes independent questions; when questions come in groups, as Arbiter's do (17 per patient record), the real uncertainty is larger, so the curve is optimistic 3.
Data
Every number in the charts, as CSV:
Agent policy
This read is evidence for these rules in the agent's policy:
Sources
- Bean et al., "Measuring what Matters: Construct Validity in Large Language Model Benchmarks", NeurIPS 2025 Datasets and Benchmarks
- Chen et al., "Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models", ACL 2026
- Miller, "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations", Anthropic, arXiv 2411.00640 (2024)
- Parupudi, "There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items", arXiv 2608.21382 (2026, preprint)
- Zhao et al., "Towards Evaluation Engineering: An Empirical Study of ML Evaluation Harnesses in the Wild", arXiv 2605.24213 (2026, preprint)
- Reuel et al., "BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices", NeurIPS 2024
- Arbiter changelog, 2026-09-08: power analysis of the eval harness
- UK AI Security Institute, "More compute, more capability" (blog, 2 Jul 2026)
- Card et al., "With Little Power Comes Great Responsibility", EMNLP 2020
Corrections
None so far. Found an error? Tell me and it will be corrected here, with the date.