Research & models Oct 29, 2026

This week · preprint, arXiv 2610.35971

Study finds mid-run coding-agent failure unpredictable

A study of 147,002 coding-agent trajectories finds that 20,000 tokens into a run, whether the attempt will fail is no more predictable than a coin flip, and in 216 of 224 configurations running to the end solved more problems with fewer tokens than restarting midway. According to the paper's abstract, authors Srinivasan, Li, Menzies and Assunção analyzed 32,406 attempts. Stopping a coding agent's run that looks bad and restarting it, to save compute, may cost more than letting it finish. The authors use AUC to measure whether, partway through, it is possible to tell which attempts at the same problem will fail, where 0.5 equals guessing; at 20,000 tokens, the within-problem AUC was 0.50 (range 0.48 to 0.51). A learned rule for when to restart solved only 1.98 percentage points more than random restarts, and the gain came mainly from not wasting restarts on easy problems. Across four settings with a 75,000-token cap, 216 of 224 configurations had a higher solve rate and lower average token use when runs continued without interruption. The abstract doesn't say which models or problem sets were used (arXiv 2610.35971). How we checked →