Benchmark saturation is what happens when the top-performing systems get so close to the maximum possible score on a test that the remaining differences between them no longer say much about real capability. The scores still exist, but they stop being useful signals.
What Benchmark Saturation Means
At its core, benchmark saturation occurs when leading systems approach the ceiling of a test. Once that ceiling is near, the gaps between scores shrink and become less informative about meaningful capability. A test that once separated strong systems from weak ones may no longer do so.
Why Saturation Matters
The problem is not that the test is broken. The problem is that the test has done its job and can no longer distinguish between systems that have all mastered it. When score differences no longer reflect meaningful capability differences, the benchmark loses its value as a tool for comparison.
Trade-offs and Evaluation
Understanding saturation involves recognizing the trade-offs. A benchmark that is too easy saturates quickly. A benchmark that is too hard may never provide useful signal. Evaluation practices must account for where a test sits on that spectrum and whether it still measures what it was designed to measure.
Controls That Matter in Practice
Controls are essential for interpreting benchmark results as saturation approaches. Without them, it is easy to mistake small score differences for meaningful capability gaps. The guide emphasizes that the mechanism, trade-offs, evaluation, and controls all matter when assessing whether a benchmark remains useful.
