The number that once anchored frontier model comparisons is no longer doing any useful work.
MMLU, the Massive Multitask Language Understanding benchmark, a 57- subject test covering everything from STEM to law and ethics, was for several years the clearest signal that a new model had crossed into frontier territory. A jump from 70 to 80 percent meant something. The benchmark separated capable models from transformational ones.
That separation is gone. Third-party leaderboard data shows top models now clustered above 89 percent on MMLU-Pro, the harder variant specifically designed to resist the saturation problem in the original MMLU. According to available leaderboard tracking, Epoch AI’s independent benchmark program confirms the clustering pattern, Gemini 3 Pro sits at 90.10 percent, Claude Opus 4.7 at 89.87 percent. Epoch AI’s benchmark data reportedly indicates the broader MMLU benchmark approaching 92 percent saturation overall. That specific figure could not be confirmed from Epoch AI’s public benchmark page during verification; it should be treated as reported, not confirmed. The directional pattern, however, is supported across multiple independent leaderboards.
The implication is blunt: if your model selection process relies on MMLU scores, you’re using a broken instrument.
Why benchmarks saturate, and why this one is saturating now
Benchmark saturation isn’t a failure of the models. It’s a success. MMLU was designed to measure a capability ceiling; when models clear it consistently, the benchmark has done its job. The problem is that the AI industry has not finished building the replacement layer fast enough.
Three dynamics are compressing the timeline. First, training data includes academic content that overlaps with benchmark question sets – top models have probably encountered MMLU-format material during training, which inflates scores relative to genuine out-of-distribution reasoning. Second, frontier compute is now reliably above the 10²⁶ FLOP threshold, the boundary that multiple independent sources, including analysis from Import AI’s coverage of compute governance, confirm as the current frontier boundary. At that scale, models have enough capacity to master structured knowledge benchmarks. Third, the reinforcement learning techniques that make models better at step-by-step reasoning also make them better at MMLU-style questions specifically.
The result: MMLU tells you that a model is frontier-class. It no longer tells you which frontier-class model is better for your use case.
What the saturation means for enterprise model selection
The enterprise buyer problem is immediate. Procurement decisions and model selection processes built around benchmark comparisons need to change.
The practical shift is toward workload-specific evaluation. A legal team selecting a model for contract analysis doesn’t need the model that scores highest on a 57-subject knowledge test. They need the model that performs best on their specific contract format, their specific legal jurisdiction, and their specific latency and cost requirements. MMLU scores were a useful proxy when the alternative was no benchmark at all. That proxy is now too coarse.
Epoch AI’s specialized evaluation suite points toward what the replacement layer looks like. The Epoch Capabilities Index attempts to score models on composite capability dimensions rather than a single aggregate score. FrontierMath, another Epoch product, tests mathematical reasoning at a difficulty level designed to remain meaningful even as general MMLU saturation accelerates. These specialized evals are the emerging standard for buyers who need a genuine signal, not a number that every top model can produce.
One practical complication: most enterprise buyers don’t have the internal capacity to run FrontierMath evaluations against their own use case. The shift toward specialized benchmarking benefits large teams and sophisticated buyers. Smaller organizations will need benchmark aggregators and third-party evaluation services to fill the gap, a market that doesn’t fully exist yet.
The regulatory dimension: why compute beats benchmarks as a governance threshold
Here’s the structural insight that connects the benchmark saturation story to the compliance landscape: the EU AI Act’s systemic risk classification is not benchmark-based. It’s compute-based.
Under the EU AI Act, a model trained above the 10²⁶ FLOP threshold triggers systemic risk obligations, enhanced documentation, safety testing, incident reporting, and cooperation with the EU AI Office. That threshold was set before MMLU saturation made the benchmark question moot for regulatory purposes. Regulators chose compute as the proxy precisely because capability benchmarks are gameable, movable, and hard to verify independently.
The compute-based threshold has its own limitations. Compute measures inputs, not outputs, a system trained with 10²⁶ FLOP isn’t automatically more dangerous than one trained with 10²⁵.⁹. But as a regulatory bright line, it avoids the benchmark gaming problem entirely. Two systems that both score 90 percent on MMLU-Pro are indistinguishable by that metric. Their training compute figures are independent facts.
For compliance teams, the saturation story reinforces one already- established principle: EU AI Act risk classification requires compute documentation, not benchmark performance reports. If your organization is building or deploying frontier-class systems, that 10²⁶ threshold is the number to track, not MMLU scores.
The independent evaluation gap
Both the enterprise selection problem and the regulatory problem converge on one gap: independent evaluation infrastructure at scale is not keeping pace with frontier model releases.
Epoch AI occupies a small but significant position as the closest thing the industry has to an independent evaluation authority. Its compute tracking, the Epoch Capabilities Index, and FrontierMath are referenced in compliance discussions, enterprise procurement processes, and governance debates. This is the fourth TJS Technology brief in seven days to draw on Epoch AI data as a primary source, not because TJS is over-relying on one source, but because Epoch AI is one of the few organizations producing independently verifiable frontier model data at all.
LMSYS Chatbot Arena, HELM (Holistic Evaluation of Language Models), and a handful of academic benchmark programs contribute to the independent evaluation layer. None of them are resourced at a scale that matches the pace of frontier model releases. When a major model releases and three tier-one journalism outlets are covering it within hours, Epoch AI is updating its database, not publishing a comprehensive independent evaluation. That evaluation comes weeks later, if it comes at all.
The practical consequence: for at least the first weeks after a major frontier model release, most published benchmark comparisons reflect vendor-reported figures. The “89.2 percent MMLU-Pro” figures you see in model cards and press releases are self-reported. Independent confirmation takes time. Enterprise buyers making near-term decisions are operating on vendor-sourced data with independent evaluation pending.
That gap isn’t closing on its own. It closes when organizations invest in independent evaluation infrastructure, when regulators require third-party testing as a compliance condition, or when market pressure makes benchmark self-reporting a credibility liability. The EU AI Act creates some of that pressure at the systemic risk tier. Below that threshold, the independent evaluation gap persists.
The benchmark ceiling is a solvable problem. The industry built MMLU-Pro when MMLU saturated. It’ll build the next layer as MMLU-Pro saturates. What the industry hasn’t solved is the lag, the period between when a benchmark saturates and when its replacement is widely adopted. Right now, that lag is where most enterprise model selection decisions live.