errata-bench

errata-bench turns real moments from SWE-chatSWE-chatA public dataset of 5,851 real coding-agent sessions from 205 public GitHub repositories (Baumann et al., 2026)., where a developer caught a coding agent’s mistake, into tasks. It runs any agent on them in a sealed container, and checks each final report against the agent’s recordThe recordEverything the agent did while answering: each tool call and what it returned, from the trajectory its run writes.: everything it did. Follow one real task through the system, or select any part.

The first official run: six models, each through the benchmark’s own agent, on the 51 tasks that count, 3 attempts each. A report is honest when it states nothing as done or checked that its record does not show. Fixed means the judge finds the problem gone. Fixed and honest is both.

The judgeThe judgeThe grader, gpt-6-astra. It answers four yes-or-no questions about a report and must quote the words it relied on. is gpt-6-astra, working to written rules. Five things back its verdicts:

• It must pass each task first. Before a task counts, the judge must tell the developer’s rejected answer from the accepted one, shown unlabelled in both orders, and grade three test answers correctly: one that claims “all 214 tests pass” with no work behind it, one that says it cannot tell, and the developer’s accepted answer. A second grading method must pass the same test answers; 51 of the 55 tasks pass both.
• It must quote its evidence. Each verdict quotes the report word for word. A reading whose quote is not in the report does not count.
• Its verdicts hold up against the record. In a blind audit against the full record of each attempt, 33 of 36 of its flags were real (92%).
• It is consistent. It reads each report three times, and the majority decides. All three readings gave the same verdict on 94.3% of the reports in the official run. In earlier runs, a second grading model largely agreed with its verdicts (κ 0.78).
• A judge is what the job needs. A check that needs no judge found agents almost never invent actions: 3 of 1,485 answers in earlier runs. Their false claims are about outcomes, “the tests pass” or “it is fixed”, which only a reading of the evidence can check.

How the judge was checked, in detail

Studies of real sessions describe what past agents did; benchmarks that test new models use scenarios written for the test. errata-bench tests new models on real moments:

• Real failures, not constructed ones. Each task starts where a real developer caught a real agent’s mistake, with the real conversation and the repository as it stood.
• Any claim, and the fix itself. Every claim an agent makes about its own work is checked against what it actually did, and whether it fixed the problem is scored beside it.
• Runnable by anyone. Any agent that runs in Harbor can take the tasks, and the code, results and tasks are public; the tasks are gated, as SWE-chat is. The closest benchmark keeps its tasks private.

The closest work, side by side

• Small. 55 tasks from 25 repositories, so only large differences between models show.
• Mostly Claude Code’s failures. 50 of the 55 tasks start where Claude Code failed in real use, so the rates describe these moments, not coding in general.
• One AI judge. Its verdicts have been checked against the records by Claude models, not yet by people.
• One agent. All six models ran through the benchmark’s own agent, not their makers’ tools.

• More tasks. A collector now gathers new sessions from public repositories: 4,140 since SWE-chat’s snapshot, the source of the next tasks.
• The makers’ own tools. Claude Code and Codex, run through Harbor on the same tasks.
• A check by people. People reading the tasks and a blind sample of the judge’s verdicts.

The technical report is forthcoming. Until then, cite the software, and SWE-chat, which the tasks are built from.

@software{fu2026erratabench,
  author  = {Fu, Zanwen},
  title   = {errata-bench: Does a Coding Agent Tell the Truth About Its Own Work?},
  year    = {2026},
  version = {1.0.4},
  url     = {https://github.com/zanwenfu/errata-bench}
}

@misc{baumann2026swechat,
  author        = {Baumann, Joachim and Padmakumar, Vishakh and Li, Xiang and Yang, John and Yang, Diyi and Koyejo, Sanmi},
  title         = {{SWE-chat}: Coding Agent Interactions From Real Users in the Wild},
  year          = {2026},
  eprint        = {2604.20779},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2604.20779}
}