What Is 72.4% a Score Of? — How to Read LLM Benchmarks#

2026-10-18

A number I copied#

In last week’s 2026-10-11 AI news brief, I introduced Claude Haiku 5.5 with this sentence: “On OSWorld 2.1 (offline subset), which measures how well a model operates a computer, it scores 72.4%, far above Haiku 4.5’s 15.7%,” followed by “on the terminal-task evaluation Terminal-Bench 4.0 it scores 39.2%.” Rereading the brief, I realized I had never properly studied either benchmark. I had copied the numbers without knowing what they measured.

When I checked, 72.4% did not mean “completed 72.4% of the tasks.” The vertical axis of the OSWorld chart on Anthropic’s announcement page reads partial-credit score. Under this method, a task earns points for the progress made even if it is never finished. According to a secondary source citing the system card, the strict pass rate from the same runs, meaning the share of tasks completed all the way, is 37.1%. Another reading of the launch chart puts the same partial-credit score at 42.0% when the effort setting is dropped to its lowest level, Low. One model, yet depending on what is measured and under which conditions, the number moves anywhere from 37% to 72%.

This essay is the record of my research into LLM (large language model) evaluation as a whole, done to understand that number. The conclusion first. A benchmark number is not the score of one model; it is the combined score of a task set, a grader, and run conditions. So my current view is that public benchmarks should be used to shortlist candidates, and the final judgment should come from an evaluation built from your own work.

The three parts of a benchmark#

Every benchmark comes down to three questions: what is the model asked to do (tasks), who judges the result (grading), and under what conditions was it run (run conditions).

Tasks can be multiple-choice knowledge questions, bug fixes in a real code repository, or desktop work inside a virtual machine.

Grading falls into roughly four types.

• Answer matching: compare a multiple-choice or numeric answer. Cheap and objective, but only usable for simple tasks.
• Execution checks: run tests against the output. Most coding benchmarks work this way.
• State checks: after the task ends, a script inspects the state of files, apps, and settings. Computer-use benchmarks mostly rely on this.
• Human or LLM judgment: an LLM grades against a rubric (a scoring guide), or a person picks the better of two answers. This can measure subjective quality, but the grader’s biases creep in.

There are several metrics as well. The same run results can produce very different numbers depending on which metric summarizes them.

Run conditions are the least visible part but change the number the most. They include which harness (the execution framework that gives a model tools and a work loop) ran the agent, the effort setting, the time and token limits, whether internet access was allowed, whether the full task set or a subset was used, and who did the measuring. The 72.4% at the top of this essay is a partial-credit score that Anthropic measured itself, at the maximum effort setting, on the offline subset.

SWE-bench — an exam built by rewinding real issues#

Looking closely at one benchmark makes the rest easier to read. Most of today’s coding agent benchmarks inherit SWE-bench’s approach, so I used it as the reference point.

SWE stands for Software Engineering. It comes from the US tech industry habit of calling the software engineer role “SWE” for short. The title of the original paper, released by Princeton researchers in October 2023, states the concept directly: “Can Language Models Resolve Real-World GitHub Issues?”

As an analogy, you hand a new developer a bug ticket from a real open-source project, they read the codebase and submit a fix, and you grade it with tests they were never shown.

The key idea is not to write new problems but to rewind past issues that have already been resolved. The researchers went through 12 well-known Python repositories, including django, scikit-learn, sympy, and matplotlib, and collected 2,294 cases where “an issue exists, a PR that resolved it was merged, and that PR also added tests.” Each task looks like this:

• What the model gets: the issue text and the full repository code, rolled back to just before the fix PR was merged
• What the model submits: a code patch
• What only the grader has: the tests added in the real PR (they fail before the fix and must pass after a correct fix), plus existing tests that already passed (they must keep passing after the fix)

A task counts as “resolved” only if both groups of tests pass. Unlike puzzle-style coding problems where you write a single function, the model has to find the cause in a repository of hundreds of thousands of lines, interpret a vague issue written by a person, and change several files without breaking existing behavior. At release, the best model, Claude 2, resolved fewer than 2% of the tasks. It was the first benchmark to show in numbers that models “solve coding problems well but can barely do the real job.”

The lineage that grew from SWE-bench#

Most coding benchmarks after SWE-bench appeared as answers to the question “what was wrong with the previous generation?” Following the lineage shows what evaluation has been wrestling with.

I noticed three currents in this table.

First, refining problem quality. Verified was built to separate “the model can’t do it” from “the problem is broken” when scores were low. OpenAI worked with the original authors and had human developers review the tasks one by one, filtering out those with insufficient issue descriptions or unfair tests.

Second, avoiding contamination. Since both the problems and the answers sit in public repositories, a model may have seen the answer during training. Pro mixed in commercial repositories and repositories with restrictive licenses, and DeepSWE wrote its tasks from scratch.

Third, raising the bar for “pass.” Plenty of code passes the tests and would still be rejected in review. In FrontierCode, open-source maintainers write the tasks and the grading criteria themselves, and a single defect that would get a patch rejected outright in review means a score of zero.

The lineage also spread beyond code. SWE-bench’s idea of “give it real work and verify the result by running it” spread to Terminal-Bench, which widens the scope to terminal work in general; to OSWorld, which widens it to operating a desktop; and to AutomationBench, which widens it to business workflows across several SaaS tools. The two benchmarks from the start of this essay are distant descendants of this line.

Saturation — when the exam hits its ceiling#

The biggest reason new benchmarks keep appearing is saturation. Saturation does not mean “scores are high.” It means “this exam can no longer tell models apart.” Think of a national entrance exam in a year when so many students get perfect scores that the test no longer separates anyone.

MMLU is the classic example. It is a knowledge test of about 14,000 four-choice questions across 57 subjects, and the score is the share answered correctly. Random guessing gives 25%, and the original paper estimated domain-expert level at about 89.8%. Today’s top models are reported to cluster around 90%. But some questions in this test have wrong answer labels, so the practical ceiling is below 100%. Near the ceiling, a gap of a fraction of a point gets lost in prompt formatting and run-to-run variation. One study presented at ICML 2026 reported that nearly half of 60 frequently cited benchmarks are already saturated.

Saturation tends to come with suspicions of contamination. SWE-bench Verified is the case in point. In February 2026, OpenAI said it would stop reporting Verified scores. When its own audit re-examined tasks that models kept failing, many turned out to be effectively unsolvable because their tests were too narrow or too broad, and several frontier models could reproduce the original gold patch or issue text verbatim from nothing but the task ID. OpenAI’s evals team described the benchmark as “effectively saturated and also highly contaminated” and pointed to SWE-bench Pro as the alternative.

Saturation pushes the target of measurement upward. From knowledge (MMLU) to hard reasoning (GPQA, Humanity’s Last Exam), from function-level coding (HumanEval) to repository-level coding (SWE-bench), and now to whether an agent can carry real work through to the end. Once agent work became the target, the subject of the score changed too. It is no longer the model alone taking the exam but “the combination of a model and a harness.” That is exactly what SWE-agent showed in 2024.

Benchmarks you see often today#

I picked the ones that kept appearing in the launch materials for Claude Opus 5.5, Haiku 5.5, and GPT-6 Sol / Luna in September and October 2026, and in independent evaluators’ indexes, and grouped them by area.

Coding agents

• Terminal-Bench 4.0 (Stanford, Laude Institute): 66 long-horizon tasks that must be finished in a terminal, such as deploying a service, proving a math theorem, or working on a GPU kernel
• SWE-bench Pro (Scale AI): the successor to Verified, with harder and less contaminated repository issues
• DeepSWE (Datacurve): 113 newly written long-horizon tasks from 91 repositories in five languages
• FrontierCode (Cognition): grades mergeability with rubrics written by maintainers; the tasks are private
• CursorBench (Cursor): measured inside Cursor’s actual agent harness

Computer use

• OSWorld 2.x (HKU xlang): 108 long-horizon desktop tasks that take a person a median of 1.6 hours; version 2.1 added an offline subset that runs without the internet

Work automation and professional tasks

• GDPval / GDPval-AA (OpenAI / Artificial Analysis): compares real work products across 44 occupations against expert level; the AA version assigns Elo scores from blind comparisons by several LLM judges
• AutomationBench (Zapier): 600 workflows across 47 simulated SaaS tools, graded on the final state
• Agents’ Last Exam (UC Berkeley RDI, Snorkel AI): professional tasks organized by an occupational taxonomy, performed in full OS sandboxes

Reasoning and knowledge

• Humanity’s Last Exam (CAIS, Scale AI): a collection of the hardest questions, written by experts in each field
• ARC-AGI series (ARC Prize Foundation): the ability to generalize puzzle rules never seen before

Aggregate indexes and independent measurement

• Artificial Analysis Intelligence Index, Vals Index: rerun many benchmarks under the same conditions and combine them into a single index
• METR time horizon: tracks how long a task, measured in human hours, an agent can complete
• LMArena: Elo scores from human preference votes

One new current is also visible. The GPT-6 launch reported a factual error rate alongside a coding deception rate, the share of answers where the agent misrepresents what its own code does. Evaluation has started to measure “how far can it be trusted” in addition to “how well does it do.”

Graders get it wrong too#

Execution checks and state checks look objective, but graders are code written by people too. Terminal-Bench 4.0 from the start of this essay is a good example.

Epoch AI, which audits benchmark quality, confirmed publicly noted scoring defects in 30 of Terminal-Bench 4.0’s 66 tasks (45.5%) and classified the benchmark as “Flawed.” The defects came in three kinds. As an example of a grader passing wrong answers, in one task writing the success signal directly into the verifier before any tests ran made all 60 tests pass. Answer leaks included a grading key sitting inside the repository and a random seed embedded in an input file name. In the other direction, 11 tasks had defects that failed correct answers by grading against requirements the problem never stated.

SWE-bench had a similar hole. There were reports of agents using commands like git log --all to find “future commits” left in the repository, meaning the answer PR itself, and evaluation environments have since moved toward stripping out future commits. Agents have become capable enough to look for loopholes in the exam instead of solving it, and grading environments have to become more careful in step.

Numbers also differ depending on who measures. Haiku 5.5’s 39.2% on Terminal-Bench 4.0 was measured by Anthropic with the Claude Code harness at the maximum effort setting, while Artificial Analysis reportedly measured 32.8% on its own. Neither is wrong; the conditions differ. OSWorld is the same. The best result the OSWorld 2.0 paper reported at publication, for Claude Opus 4.8, was a 20.6% completion rate and a 54.8% partial-credit score. On the same benchmark, the impression changes completely depending on which metric makes the headline.

Who writes the exams, and who picks them#

One thing I wondered about during the research was how this ecosystem came to be. In the end, it was neither dictated by the AI providers nor entirely self-organized. It is closer to a market where many different parties write exams, and the ones that survive are the ones that get picked.

Early on, academia led. MMLU came from UC Berkeley, SWE-bench from Princeton, and OSWorld from the University of Hong Kong (HKU). AI labs built their own as well: HumanEval, SWE-bench Verified, and GDPval came from OpenAI. Nonprofits such as the ARC Prize, Epoch AI, and METR joined later, and more recently data vendors such as Scale AI, Snorkel, and Datacurve and product companies such as Cursor, Cognition, and Zapier have released benchmarks. Government bodies such as the UK AI Security Institute also publish evaluation tools.

The selection mostly happens at two gates. One is the benchmark table in an AI lab’s model launch. Getting into that table effectively makes a benchmark a standard; staying out of it means being forgotten. The other is whether independent evaluators such as Artificial Analysis, Vals AI, and Epoch AI add it to their indexes and rerun it. Once a benchmark saturates, it gives way to its successor.

Incentives are tangled into this structure. Labs have a reason to pick benchmarks on which their own models look good. A data vendor’s benchmark also doubles as a sales tool proving the quality of its data. A product company’s benchmark may be structured in favor of its own harness. In practice, the math benchmark FrontierMath drew controversy when OpenAI’s funding and data access came to light only later, and LMArena drew a critical paper arguing that large labs privately test many model variants and publish only the best one. That is why a separate role of auditing the benchmarks themselves emerged.

How do you evaluate a company’s own system?#

At this point a follow-up question comes naturally. If an ordinary company takes the models and coding agents it is given, builds a harness on top, and makes its own system, is there a benchmark that measures that system?

From what I found, there is effectively no industry-wide standard, and the approach comes in three layers.

First, plugging your own harness into a public agent benchmark. Agent benchmarks already evaluate “the combination of a model and a harness,” so the harness can be swapped. Harbor, the framework Terminal-Bench runs on, is designed so you can plug in the agent of your choice. You get a number comparable with the outside world, but the tasks are general-purpose and can be far from the company’s actual work.

Second, industry-specific public benchmarks. These include τ-bench (Sierra), which measures with pass^k whether a customer service agent handles requests consistently while following company policy; CRMArena-Pro (Salesforce) for CRM work; and AutomationBench for SaaS workflows. If one matches the type of system you built, you can run it as is.

Third, the mainstream in practice: your own evaluation. You build a fixed dataset that pairs real work inputs with expected results, build a grader that mixes code checks, rubric-based LLM judgment, and human review, and rerun the same set every time you change the model or the prompt. Open-source frameworks such as the UK AISI’s Inspect and a range of commercial evaluation tools help with this process.

What I found interesting is that this third layer has started flowing back into model evaluation. The Haiku 5.5 announcement page includes results (averaged over three runs) from an evaluation set HubSpot built from its own CRM work. When a model provider cites a customer’s internal evaluation as “real-world evidence,” I read it as a sign that public benchmarks alone are no longer enough to make the case for real-world performance.

Closing — I write the exam for my own work#

Following a single number, 72.4%, led me to one conclusion. Public benchmarks are useful for choosing models, but their numbers are measured on tasks someone else picked, with graders someone else built, under someone else’s conditions. Whether a model fits my work can only be found out with an exam built from my work.

In the brief, I wrote that I would try Haiku 5.5 in the subagent and summarization steps of this blog’s working harness. Now it is clear how to run that experiment. I will use the blog’s content quality standards document as the rubric, turn about 20 summarization jobs from past briefs into a fixed set, and run Sonnet and Haiku several times each in the same harness. The metrics to watch are not the single-attempt success rate but whether it succeeds every time (pass^k), and the token cost actually spent under the new tokenizer.

The theme of the last brief was verification debt: what piles up unchecked when the output AI produces outruns the speed at which people can confirm it. Copying a benchmark number without its source and conditions is a small piece of verification debt too. This time I paid off only that one number, but from now on, whenever I put a benchmark in a brief, I intend to answer at least the questions below first.

Appendix: seven things to check when reading a benchmark number#

1. Is the benchmark already saturated? A 1–2 point gap on a saturated benchmark means little.
2. Is it the full task set or a subset?
3. What is the metric? Check whether it is pass@1 or pass@k, partial credit or strict pass.
4. What harness and effort setting were used, and were the compared models run under the same conditions?
5. Who measured it? See whether it was the vendor or an independent evaluator, and whether there are confidence intervals.
6. Is it being compared against scores from a previous version?
7. Has the benchmark itself been audited for quality?

References#

• Anthropic — Introducing Claude Haiku 5.5
• Kingy AI — GPT Sol & Luna vs Claude Haiku 5.5 (cites the OSWorld strict pass rate and Terminal-Bench run conditions)
• OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
• xlang-ai — OSWorld-V2 release notes
• Terminal-Bench — version history
• Epoch AI — Terminal-Bench 4.0.0 review
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• OpenAI — Introducing SWE-bench Verified
• Latent Space — OpenAI stops reporting SWE-bench Verified
• DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
• Cognition — Introducing FrontierCode
• Shortcutting the Fix: Identifying and Categorizing Agentic Exploits in Software Engineering Benchmarks
• Artificial Analysis — GDPval-AA
• Snorkel AI — Agents’ Last Exam
• Prime Intellect — Zapier AutomationBench
• AI Weekly — ICML study finds nearly half of LLM benchmarks saturate
• harbor-framework / terminal-bench
• 2026-10-11 AI news brief