Research & models Oct 28, 2026

Trend · first published August 13, 2026

Hugging Face says human-led paper checks were most reliable

In the ICML 2026 community reproduction run by Hugging Face and alphaXiv, 1,221 people with coding agents checked 2,226 papers in 19 days, and the organizers say the most reliable results came from human-directed work. The original post is from August 13. It appears here because its conclusion points the same way as today's Epoch item. Hugging Face hosts open models and datasets, and alphaXiv is a platform for discussing papers. The 1,221 participants, each working with a coding agent, published 6,816 reproduction logs from July 15 to August 2, covering about a third of ICML 2026's accepted papers and ruling on 35,908 individual claims. Of the 2,226 papers, 1,103 had at least one claim independently verified and 496 had at least one claim refuted or disputed; the same paper can be in both groups. In 242 of those 496, different teams reached opposite rulings on the same claim. For one spotlight paper, participants found the step where its proof breaks, and the organizers confirmed it by redoing the work at larger scale. The organizers concluded that the most reliable results came from human-directed workflows, in which people redirect the work, question assumptions and spot a wrong experimental premise before compute is burned. The limits: the organizers report the results themselves, participants chose the papers, rulings were made automatically by one open-weight model, GLM-5.2, and the percentages hold only for the third of papers that were attempted (Hugging Face, 2026-08-13). How we checked →