Model watch · Today (submitted August 24)
From the Aug 25, 2026 daily brief
The standard way to evaluate code-rewriting ability checks only whether behaviour is correct, never whether the migration actually happened — which leaves one very simple cheat available: copy the original implementation across so the tests pass. A paper posted to arXiv, the open preprint platform, yesterday names that failure Blindness. It builds a three-stage evaluation around it: first verify the migration occurred; then measure behavioural correctness against a fixed test suite; then set 6 independent coding agents to generate targeted tests hunting hidden behavioural differences. The population is 20 whole-repository migration tasks covering 4 kinds of technical debt, run across 26 configurations of 8 frontier models, 520 runs in total. Only 28 of them (5.4%) cleared all three stages, 13 of the 20 tasks received no accepted solution at all, and the best-performing model, claude-opus-5, scored 47.0 on a 100-point scale (arXiv 2608.23564, 08-24). The most informative part is how the runs failed. A few skipped the migration and fell at the first stage, but the overwhelming majority genuinely attempted it and then broke behaviour, falling at the second. ⚠️ Two qualifiers: this is an evaluation, not a real migration (it involves no cross-team work and no production cutover, and it says nothing about defect rates after launch), and it is a preprint nobody has peer-reviewed. It cannot tell you how large the bill for moving house is. It answers the other half: today the move mostly cannot be made, and the main way it fails is by breaking things. That is exactly the shape of the repair breaking your own service in the expected-value argument in main line item 2.
Just an email address, unsubscribe anytime. This is the only thing we ask of you.
What follows is not a preprint. It is a set of readings from the appendix of the measureme…
The instrument here belongs to someone else — an evaluation called CoT-Control, which appe…
What follows is not an arXiv preprint but a research team's own write-up of its own paper …
The CAI team at Multiverse Computing, writing up its paper on Hugging Face, reports that a…