Looking back · papers from 2023 to 2025

Three research groups found that reward models and AI judges favor longer answers.

Concept illustration of tilted, overlapping paper sheets on a beige backdrop: on the large left sheet, two framed pages hang at the top, the left with gray lines and the right with gray lines plus a plain lower section attached by two tabs, and below them a black screen lettered "REWARD MODEL" stands beside a tall stack of red sheets and a shorter stack of beige sheets, each on a tray with a paper slip showing at the front, while a person in black seated on a stool holds a sheet with gray lines on top and a plain attached lower section over the beige stack. On the sheet to the right, a black screen lettered "AI JUDGE" sits on a shelf next to a tray holding an upright long sheet made of a lined top and a plain lower part joined by two tabs, and a smaller lined sheet, while another sheet with gray lines shows at the bottom right. Printed on the image: 60.1% Base reward models picked longer, self-reported.

According to the authors' self-reported experiments, preference scores and model rankings produced by AI judges lean toward longer answers unless answer length is controlled. A 2023 paper found that in three RLHF settings, most of the rise in reward scores could be explained by answers getting longer (arXiv 2310.03716). The 2024 AlpacaEval paper found that controlling for length raised the correlation between AlpacaEval, an automatic scoring system that uses an AI as judge, and Chatbot Arena from 0.94 to 0.98 (arXiv 2404.04475). Chatbot Arena is a public leaderboard built from users' votes between two anonymous models' answers. A 2025 paper ran a control that only lengthened answers without adding content: base reward models picked the longer version 60.1% of the time, against 33.7% for humans. The same paper found that when GPT-4o, Gemini 2.5 Pro and Claude 3.7 Sonnet acted as judges, length and vagueness were where they departed most from human preferences (arXiv 2506.05339). These experiments tested only the judge models of their time, and no one has directly measured the effect on leaderboard rankings. How we checked →

Source and checks

This item ran in the brief of Nov 10, 2026, alongside 12 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.

Cite this item

SecondSource (2026). "Three research groups found that reward models and AI judges favor longer answers.." https://secondsource.io/en/research/2026-11-10-2/. Accessed access date.