Today · preprint, October

ETH finds agent tests miss planted bugs; it's one preprint

A preprint from an ETH Zurich team says coding agents build software from specs, and that software passes 82% of the reference tests. But the tests the agents write themselves catch fewer than 60% of deliberately planted bugs. Across 165 applications, the researchers broke each one in 74 places on average and checked how many of those breaks the agents' tests caught. The paper's conclusion is that "agents can build software they cannot verify" (TestSuiteBench preprint). The researchers also report that 88% of the agents' tests pass on correct code. So the tests aren't wrong. They are too weak to catch errors. We read only the abstract. We don't know which agents were tested, or whether these figures are averages or the best single agent. Before you accept an agent's tests as your acceptance check, you can deliberately break the code in a few places and see how many breaks the tests catch. This check is our suggestion. The paper didn't run it. How we checked →

Concept illustration: on the left, a laptop sits on a long table. At the top of its screen is a box of grey lines, below it are rows of grey lines with three of them in red, and a narrower column of short grey lines runs down the right side. On the right, a person with long dark wavy hair, a black top, wide black trousers, white sneakers and a lanyard with a white badge sits cross-legged on a stool with one hand on the laptop keyboard, looking at the screen, in front of a pale wall with a light pink vertical band and a beige sign lettered "ETH ZURICH".

Source and checks

This item ran in the brief of Oct 13, 2026, alongside 13 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.

Cite this item

SecondSource (2026). "ETH finds agent tests miss planted bugs; it's one preprint." https://secondsource.io/en/research/2026-10-13-2/. Accessed access date.