Research & models Oct 29, 2026

This week · preprint, arXiv 2610.37283

Stuttgart's BenchMeUp validated 91% of the tasks it built

The University of Stuttgart's BenchMeUp uses an agent pipeline with a verification loop to build software engineering benchmark tasks from natural-language specifications; the authors checked 91% of the 198 tasks it built as fully valid, against 5% of 200 targets for the general-purpose agent Codex run as a control. According to the paper's abstract, authors Islem Bouzenia and Michael Pradel chose four task types, program repair, issue reproduction, compiler autotuning and fixing static-analysis warnings, with 50 targets each, and ended up with 198 tasks. Our inference: if BenchMeUp's reported average of US$0.79 per task holds beyond the authors' four task types, teams building their own evaluation tasks could do it for under a dollar each. BenchMeUp took an average of 36 minutes per task. The authors checked source code, specification and expected results one by one, and 180 of the 198 tasks (91%) were fully valid. The control was Codex, a general-purpose coding agent the authors ran separately against the same 200 targets, which delivered only 10 valid tasks (5%). The abstract doesn't say whether Codex had the same specifications, budget and verification loop as BenchMeUp, so the 91% against 5% can't be credited to the verification loop alone. The authors also write that running is not the same as being semantically correct (arXiv 2610.37283). How we checked →