Looking back

Harvey links scores to data-room reading; it's a self-report

(SecondSource deep dive of October 8, on a September 8 experiment by Harvey and Baseten) In Harvey and Baseten's test, long-document AI scores rose with how much of the data room the system read. Across seven models, a recursive harness averaged 62.4% on a simulated M&A due-diligence task, against 23.3% for a simple tool loop. A data room is the large set of documents used in M&A due diligence. In a recursive harness, a lead agent writes code to split the data room into chunks and hands each chunk to a sub-agent. Legal AI company Harvey and inference platform Baseten found that the simple tool loop never read more than 1% of the data room at a time. The recursive harness mostly came close to reading all of it, and the more it read, the higher it scored. The deep dive rechecked the claims sentence by sentence and narrowed two of them. First, "general-purpose agents are the wrong default" holds only under Harvey's minimal instructions. Claude Code and Codex can dispatch sub-agents, yet neither did. An independent academic paper also reached a finding that runs against Harvey's claim. Second, Harvey's own per-model chart shows a post-trained open-source lead agent scoring 63.0%, below the 77.6% of Opus 5 without post-training. Beyond the harness, the model itself affects the score. The numbers are a single self-report from Harvey, graded by a model. They are research readings, not deployment readings (SecondSource deep dive, 2026-10-08). Our reading: long-document AI scores should be read alongside the share of the data room read per pass. If Claude Code or Codex, rerun on a similar data room with sub-agents on, still trails a purpose-built harness by more than 20 points, then "the gap is mainly about not reading everything" doesn't hold. That 20-point threshold is SecondSource's own. How we checked →

Concept illustration divided into three framed bands: at the top, a black H-shaped mark and a green striped mark are joined by short lines to the words HARVEY and BASETEN respectively; in the middle band, a white plate lettered TOOL LOOP is joined by a thin line to a single black laptop whose screen shows a speech bubble of grey lines, sitting on an open red binder, beside a long brown-framed shelf holding a row of upright white binders with one gap in the middle. In the bottom band, a white plate lettered RECURSIVE HARNESS is joined by a thin line to the first of four black laptops in a row, each sitting on an open red binder, the first screen showing a layout of grey lines and the other three showing speech bubbles, beside a shorter brown-framed shelf that holds only two white binders at its right end.

Source and checks

This item ran in the brief of Oct 20, 2026, alongside 11 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.

Cite this item

SecondSource (2026). "Harvey links scores to data-room reading; it's a self-report." https://secondsource.io/en/news/2026-10-20-4/. Accessed access date.