A preprint recomputes public scores on eight agent benchmarks and says the crowding at the top of the leaderboards comes from how they are scored.

Concept illustration: a beige tabletop is split down the middle by an upright grey divider. On the left half, a white sign on a stand reads "PASS-FAIL", with a red laptop and a grey laptop side by side at the back; on the right half, a white sign on a stand reads "CONTINUOUS SCORE", with a grey laptop at the back right and a red laptop further forward near the front edge. Every laptop screen shows a speech bubble with grey lines, and a sheet of paper with a heading bar and two columns of lines lies on a white surface in the lower-left corner. Printed on the image: Over 10x Preprint's Terminal-Bench rerun: pass/fail-tied models differ on a quantity scoring doesn't record.

According to the paper's abstract, gaps between adjacent agents in SWE-bench Verified's top ten are smaller than the test's own error, so rank differences among the top ten may fall within noise. The three authors identify two failure modes. The first is recording each run only as pass or fail. The second is scores capped by a reference value. ALE-Bench is an agent benchmark built on algorithm-contest problems; its top ten separate on contest scores but all tie when scored against the human median. In a Terminal-Bench rerun, models tied on pass or fail differed by more than ten times on a quantity the pass-or-fail scoring doesn't record. Against today's pass-or-fail scoring, the authors propose continuous scores. No other team has adopted this scoring yet, and the abstract doesn't say how the test error was estimated or which quantity Terminal-Bench was compared on (arXiv 2611.13541, 2026-11-10). How we checked →

Source and checks

This item ran in the brief of Nov 11, 2026, alongside 15 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.

Cite this item

SecondSource (2026). "A preprint recomputes public scores on eight agent benchmarks and says the crowding at the top of the leaderboards comes from how they are scored.." https://secondsource.io/en/research/2026-11-11-1/. Accessed access date.