Research & models Oct 21, 2026

Trend · research note October 20

CRUX authors reject both AI papers; only two were tested

Princeton CITP researcher Sayash Kapoor publishes a follow-up to the CRUX evaluation: Fable 5.1 and GPT-6 Astra each wrote an AI research paper on a US$10,000 budget per model, and the original authors rejected both. CRUX evaluates AI agents doing open-ended AI research. The July round used Opus 4.8 and GPT-5.6 Sol, gave them the research questions of two unpublished papers and a few thousand dollars of compute, and the original authors rejected both papers using the NeurIPS review template. This round, Kapoor writes, "we observed meaningful improvements to the agents' capabilities at experimental design and resource management," while judgment and the calibration and expression of confidence still fall short. Calibration means stated confidence matches actual accuracy. Both papers were rejected again (Sayash Kapoor, 2026-10-20). That is two papers from one team. The progress could also come from the bigger budget: this round gave each model US$10,000, while the July tweet said only "thousands of dollars," which SecondSource logged as about US$3,000. The full research note hasn't been reviewed. Our take: over two rounds, experimental design and resource management improved, and research judgment still hasn't passed. This round changed both the models and the budget, so which one drove the progress can't be separated, and each round had only two papers. Separately, arXiv restricted access last night, so all 50 new papers are unread. Whether a major new paper appeared this week is therefore unknown.