This week · preprint, arXiv 2610.39641

Samyama AI says only 9 model rulings were correct

Two authors at Samyama AI, which builds research knowledge graphs, hand-checked 41 model rulings and found 9 fully correct. Having a model rule automatically that a paper solved an open problem isn't reliable yet, according to the authors' self-built audit, which no other team has repeated. Madhulatha Mandarapu and Sandeep Kunkunuru published on October 29, building a knowledge graph of 2,053 open problems across database research and AI research. The 41 rulings fall into two types: 34 said a problem had been refuted, and 7 said one had been solved. Checked by hand against the papers, 9 were correct, 30 partly correct and 2 wrong. The authors separately counted a stricter result: none of the 34 truly refuted an open problem, and only 1 of the 7 truly solved the core problem. The part of the paper reporting these counts doesn't split the 9 correct rulings between the two types. The authors' design conclusion is that a status change may only be proposed with evidence and never applied automatically (arXiv 2610.39641). How we checked →

Concept illustration of a large open book with two people in black turtlenecks, one with short curly hair and one with long hair, standing at its center and reading the sheets of paper they hold; at the lower left page are a laptop showing a speech bubble and a stack of paper with a sheet lettered VERDICTS leaning against it. On the right page, a black banner at the top carries a blue-and-orange looped mark and the lettering SAMYAMA AI, and below it, from left to right, are a short red stack of paper, a tall white stack whose top sheet has a torn corner, and two cards each showing a black beetle.

Source and checks

This item ran in the brief of Nov 1, 2026, alongside 7 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.

Cite this item

SecondSource (2026). "Samyama AI says only 9 model rulings were correct." https://secondsource.io/en/research/2026-11-01-1/. Accessed access date.