Research & models Aug 25, 2026

Today · submitted August 24

A new public benchmark measures whether AI can complete a whole-repository technical migration on its own: across 520 runs, 5.4% cleared all three stages.

The standard way to evaluate code-rewriting ability checks only whether behaviour is correct, never whether the migration actually happened — which leaves one very simple cheat available: copy the original implementation across so the tests pass. A paper posted to arXiv, the open preprint platform, yesterday names that failure Blindness. It builds a three-stage evaluation around it: first verify the migration occurred; then measure behavioural correctness against a fixed test suite; then set 6 independent coding agents to generate targeted tests hunting hidden behavioural differences. The population is 20 whole-repository migration tasks covering 4 kinds of technical debt, run across 26 configurations of 8 frontier models, 520 runs in total. Only 28 of them (5.4%) cleared all three stages, 13 of the 20 tasks received no accepted solution at all, and the best-performing model, claude-opus-5, scored 47.0 on a 100-point scale (arXiv 2608.23564, 08-24). The most informative part is how the runs failed. A few skipped the migration and fell at the first stage, but the overwhelming majority genuinely attempted it and then broke behaviour, falling at the second. ⚠️ Two qualifiers: this is an evaluation, not a real migration (it involves no cross-team work and no production cutover, and it says nothing about defect rates after launch), and it is a preprint nobody has peer-reviewed. It cannot tell you how large the bill for moving house is. It answers the other half: today the move mostly cannot be made, and the main way it fails is by breaking things. That is exactly the shape of the repair breaking your own service in the expected-value argument in main line item 2.