Research & models Oct 29, 2026

This week · preprint, arXiv 2610.37459

M4Bench finds specs, not skill packs, drive agent gains

On an eight-task clinical subset, M4Bench finds a complete specification, not a skill pack, drives most of an agent's score gain. The specification adds 0.162, and a skill pack adds another 0.024, which is statistically indistinguishable from zero. According to the paper's abstract, the work was accepted to the NeurIPS 2026 datasets track, the authors include MIT's Leo Anthony Celi, and the tasks are derived from clinical electronic health records. On these clinical data tasks, most of the agents' extra score came from complete specifications, and the skill packs' own contribution was too small to measure. The score here is partial agreement between the agent's answer and the reference answer. On the eight-task controlled subset, the specification alone raised the score by 0.162, and adding a skill pack raised it only another 0.024, statistically indistinguishable from zero. The subset has only eight tasks, so failing to measure a difference doesn't establish that there is none. A skill pack filled with fake content scored the same as no skill pack. Giving the agent the original reference SQL as its skill pack scored almost the same as a purpose-written skill pack. A second metric is the number of runs passing every strict check: 25 of 280 runs with a skill pack passed, against 13 of 280 without. The abstract doesn't say which model was used or whether that gap is significant (arXiv 2610.37459). How we checked →