Research notes · Retrospective (June 2024 interview)
From the Jul 20, 2026 daily brief
François Chollet, creator of the Keras framework, once gave the claim "large models memorize rather than reason" a testable probe: models can solve a Caesar cipher — the beginner's encryption that shifts every letter forward by n places — but mostly only when n is 3 or 5, the values most common in online tutorials; move it to 9 and they fail. If a model were genuinely synthesizing the solution on the spot, the shift size would not affect difficulty at all (Dwarkesh Podcast, Jun 2024). The tension when set beside today's lead 1: the 2024 diagnosis was "no on-the-fly synthesis of algorithms," and 2026 has produced a perfect-score formal-proof claim. In fairness, the tasks differ in kind — the probe tests instant synthesis inside the model's parameters, while the IMO system is a multi-agent assembly wrapped around a Lean toolchain — and June 2024 predates the reasoning-model era. The probe's current status has no public retest; we log it as pending.
Just an email address, unsubscribe anytime. This is the only thing we ask of you.
What follows is not a preprint. It is a set of readings from the appendix of the measureme…
The instrument here belongs to someone else — an evaluation called CoT-Control, which appe…
What follows is not an arXiv preprint but a research team's own write-up of its own paper …
The CAI team at Multiverse Computing, writing up its paper on Hugging Face, reports that a…