Long-document AI scores track how much it reads: in Harvey's diligence test, a recursive harness lifted seven models' average pass rate 39 points over a simple tool loop

Key takeaways

Harvey, a legal AI company, and Baseten, an inference platform, published a set of experiments on September 8. The two companies built more than a hundred simulated M&A due-diligence data rooms. A single room holds up to 5,000 documents and about 80 million tokens, far more than any model can take in at once. The same seven models were then run inside a "recursive" harness: a root agent writes code to split the data room by category, hands each slice to one of many sub-agents to read, and compiles what comes back into a memo. Across the seven models, the average pass rate rose from 23.3% in a simple tool loop to 62.4% in the recursive harness. The pass rate is the share of items on an expert checklist that a model acting as judge marks as passed in the memo. Two off-the-shelf general coding agents did worse than the models underneath them. Claude Code scored below Opus 5, and Codex below GPT-5.6 Sol, when each model ran in the simplest tool loop.

This publication earlier made a call based on that post. This piece calls it the original call. The original call has five sentences. 1: the same models gain 39 points on average when moved into a task-specific recursive harness. 2: off-the-shelf Claude Code and Codex lose to the simplest tool loop. 3: treating a general coding agent as the universal harness is the wrong default. 4: switching the harness pays more than switching to a frontier model. 5: the defensible position for a vertical AI company is a harness built for the task plus an open-weight model post-trained around it. This piece re-tests each sentence. Sentences 1 and 2 hold as numbers but need their test conditions attached (see "What Harvey measured" and "General agents lose on defaults, not on ability"). Sentence 3 has to be narrowed (see "General agents lose on defaults, not on ability"). Sentence 4 is half right (see "Harness beats model: half right"). Sentence 5 does not stand (see "What vertical AI can still defend"). The conclusion comes in three layers.

First, the mechanism holds, and it is more specific than the original sentence: the gap comes from how much gets read. Harvey used probes to measure how much of each data room actually reached the model in each run. The simple tool loop never read more than 1% of a room, and most runs read between 0.1% and 0.5%. The recursive harness mostly read close to all of it. The more a run read, the higher it scored.

Second, sentence 3 of the original call, that general agents are the wrong default, does not survive as worded. It holds only under Harvey's setup. Harvey gave Claude Code and Codex the same minimal instruction as the tool loop. The traces show both agents stopping their reading early. Neither spawned sub-agents, although both can. An academic measurement unrelated to Harvey points the other way. A March paper by four academic researchers reports that off-the-shelf coding agents beat the published best results on several long-context benchmarks by 17.3% on average. That paper does not state whether the figure is a relative gain or percentage points. As relayed by a second paper, the coding agents also beat the recursive harness on one of those benchmarks. In May, Anthropic built into Claude Code the ability to spawn tens to hundreds of sub-agents at once.

Third, the half of the original call about post-trained open-weight models is the weakest. Harvey's own per-model chart puts the post-trained open-weight root agent at 63.0%. That is below Claude Opus 5 without post-training at 77.6%, below Grok 4.5 and GPT-5.6 Sol, and below the open-weight GLM-5.2 in the same harness. After the harness is fixed, the choice of root model still moves the score by about 35 to 38 points, the same order of magnitude as the harness effect. Harness and model complement each other.

What this piece tests: the core numbers behind the original call come from Harvey alone. The two sentences with the biggest consequences get tested here. Sentence 3, that general agents are the wrong default, is tested against an independent paper and does not survive. It narrows to this: general agents run on default settings read too little of a long document. Sentence 5, that a post-trained open-weight model is a defensible position, is tested against Harvey's own per-model chart and does not survive either. This piece also states up front what result would make it wrong. Suppose someone reruns Claude Code or Codex on the same kind of data room with sub-agent delegation switched on. If the general agent still trails the task-specific harness by a wide margin, say twenty points or more, then this piece's claim fails: that the gap is mainly unfinished reading. The twenty-point threshold is this publication's own.

The piece runs in five steps: ① what Harvey measured; ② scores track how much got read; ③ general agents lose on defaults, not on ability; ④ the claim that harness beats model is half right; ⑤ what vertical AI can still defend, and which readings to watch next.

What Harvey measured

The design is clean: models, data rooms and grading stay fixed, and only the harness changes.

Three harnesses:

• Simple tool loop: the model searches on its own, picks documents to read, and writes when it has read enough. This is the standard setup in Harvey's legal agent benchmark.
• Recursive harness: the whole data room sits inside a programmable environment. The root agent does not read documents directly. It writes code to split the room by category, hands each slice to a sub-agent to read, and the sub-agents store their findings for the root to compile into a memo. The design comes from a paper by MIT's Alex Zhang and co-authors late last year, called Recursive Language Models. The method lets a model work on text far longer than its context window by operating on that text through code and handing pieces to further model calls.
• Off-the-shelf general coding agents: Claude Code with Opus 5 and Codex with GPT-5.6 Sol, with web access switched off.

Grading is done by a second model acting as judge. The judge checks the memo item by item against a checklist written by experts; the example data room's checklist has 571 items. The per-model scores follow. This publication read the labels off Harvey's chart image, and the averages match Harvey's text:

Harvey separately placed the open-weight model Qwen into the recursive harness as the root agent and post-trained it. On 50 held-out data rooms the score went from 29.9% to 63.0%. That experiment and the GLM-5.2 self-fine-tuning described under "General agents lose on defaults, not on ability" are two different experiments.

The off-the-shelf agents ran with a minimal instruction and web access off, as described under "General agents lose on defaults, not on ability". Claude Code scored 24.6%, which is 17.9 points below Opus 5 in the simple loop. Codex scored 12.0%, which is 4.6 points below GPT-5.6 Sol in the simple loop.

Three things frame this table. The data rooms are simulated, not real transactions. The grader is a model, and Harvey has not said how closely that grader agrees with lawyers. The reward used in post-training is the score from that same judge, so the examiner's standard is also the training signal. Harvey calls the training experiments "initial explorations". The post makes no claim of product deployment, and it says plainly there is "still a substantial gap between these scores and the near-perfect pass rates we are working toward". So this is a cleanly designed research reading, measured by one company.

Scores track how much got read

With only the harness changing, the next question is why that change alone would add nearly forty points. Harvey's answer is direct: the amount read differs by tens to hundreds of times.

"In the standard tool-loop harness, no run reads more than 1% of the data room, and most read between 0.1% and 0.5%. In the RLM harness, nearly every run reads more than 10% of the data room and most read close to all of it." And: "Higher coverage is associated with higher rubric criteria pass rates across this range."

The way diligence is graded makes this close to inevitable. Some checklist items can only be found by combining several documents. Some require confirming that a document which should exist is absent. An item about absence can only be proven by finishing the reading: a cabinet nobody opened cannot be declared empty. An agent that reads selectively cannot score on that class of item.

The same logic explains a counterintuitive result in Harvey's appendix. Adding one more layer of recursion, so that sub-agents can delegate in turn, made ten of fourteen data rooms worse and cut the average by 19 points. In four of those ten, the agent read the material but never produced a report. An independent reproduction paper in March saw the same cost of deeper recursion on other benchmarks: a second layer lowered scores and inflated execution time "from 3.6s to 344.5s". Another research group's paper tested a method that drops recursion and instead has the model inspect its own work and choose how to process the text. Under the same time budget, that method did better than the recursive harness by "up to 22%". The abstract does not state whether that figure is relative or in points.

Taken together, these three results say recursion itself is not a necessary ingredient, and extra layers can do harm. What actually moves with the score is how much got read. Harvey measured that higher coverage goes with higher pass rates. Harvey reports the two only as associated, though, and has not ruled out a third explanation: stronger delegation could produce both more reading and higher scores.

For teams building long-document agents, these readings suggest diagnosing in a fixed order. First check how much of the material the agent actually read and whether it really delegated. Only then tune the model or the tools.

General agents lose on defaults, not on ability

Reading coverage also explains the most striking sentence in the original call, that off-the-shelf Claude Code and Codex lose to the simplest tool loop. The numbers are real. What they measure needs stating.

Harvey is explicit. The two general agents were run "using the same minimal instruction as the tool-loop harness", and in the traces "both agents stop reading early and write shorter memos, and neither spawns sub-agents despite having the ability to." The two agents lost the same way the previous section describes: by not finishing the reading. That is a reading of default behavior, not of the ceiling of ability.

Harvey's own data contains a control. The open-weight GLM-5.2, placed in the recursive harness as root, also started out with frequent runs where the root "gives up early, under-delegates, or fails to convert the subagent outputs into a strong deliverable". Harvey selected a set of GLM-5.2's own good, high-coverage traces and fine-tuned the same model on them. In a small experiment on a separate set of twenty data rooms, the score went from 46.1% to 60.1%. The number of sub-agents spawned began to track the size of the data room, with the correlation rising from 0.17 to 0.84. Those rooms differ from the table above, so the figures are not directly comparable. Harvey's conclusion is that the behaviors a strong root agent needs, such as delegating the full review, "are learnable from a relatively small number of on-policy traces, with no privileged information and no labeled legal knowledge."

That sentence shakes the original call most. The delegation habit that lets the recursive harness pull ahead needed no labeled legal knowledge in Harvey's experiment. Our read: a general-purpose agent should in principle learn the same habit. Harvey has verified it on GLM-5.2 alone. What is visible today is that model vendors have put large-scale delegation into their general agents. A measurement of that capability on long-document review is still missing:

• Anthropic on May 28 launched dynamic workflows in Claude Code, now generally available. With the feature, Claude writes its own orchestration scripts that "run tens to hundreds of parallel subagents in a single session". The official examples are all coding tasks. No measurement of the feature on long-document review has appeared in the material reviewed for this piece.
• An independent academic measurement points the other way. A March paper by four academic researchers puts long texts into a file system and lets the latest off-the-shelf coding agents search them with their usual file tools. Across several long-context benchmarks, the agents beat the published best results by 17.3% on average. The paper does not state whether that is a relative gain or percentage points. A June paper relays the March authors' numbers on the Oolong benchmark. Oolong is a synthetic test of aggregating facts across a long text, not legal documents. The relayed numbers: coding agent 71.75%, recursive harness 64.38%, the whole text fed straight into the model 59.22%. The June paper then let the coding agent spawn full sub-agents in parallel, which raised the score to 81.36%. A reading in the opposite direction also exists. The original Recursive Language Models paper self-reports a 13% lead over Claude Code on the median of four long-context tasks. That paper came out earlier and compared against the Claude Code of that time. Results vary by benchmark.

So the fairer statement is this: on long-document work, a general agent run on default settings reads too little. Once it is made to delegate, it does not necessarily lose, and on another benchmark it has won. Harvey did not test a general agent with sub-agent delegation switched on. That is exactly the condition this piece writes down as what would prove it wrong.

Harness beats model: half right

If the general agent's loss is a matter of defaults, the next sentence to test is the original call's ranking of harness above model. The original call said the gain from the recursive harness is larger than the gain from switching to a frontier model. Checked against Harvey's own chart:

• On average, the harness effect is indeed large. The same model gains 39.1 points on average from the harness change. Inside the simple loop, the best and worst models differ by 27.4 points (Opus 5 versus GLM-5.2).
• Once the harness is fixed, the model gap does not disappear. Inside the recursive harness, the best and worst root agents differ by 35.3 points (Opus 5 at 77.6% versus DeepSeek V4 Flash at 42.3%). Harvey also ran a cross experiment, four root models each paired with three sub-agent models, to see which layer decides the score. Changing the root moved the score by about 38 points on average; changing the sub-agents moved it by about 8. Token use runs the other way: the root itself consumes very little. With Opus 5 as root, the root accounted for 3.8% of input tokens and 1.1% of output tokens.

Cheap small models can do the document reading. The model matters most in the root agent, which decides how to split the room, whether to finish reading, and how to turn several hundred findings into one memo. The best score comes from the strongest model inside the right harness.

Cost is also less one-directional than the original call implied. Harvey states that the recursive harness raised cost for six of the seven models. The following are approximate values this publication read off Harvey's log-scale chart. Cheap models went from under one dollar per data room to about US$3 to US$6, and GPT-5.6 Sol to about US$27. The one exception is Opus 5. In the simple loop, where it read the documents itself, Opus 5 spent about US$18 per room. As root agent it spent about US$7 and scored 35 points higher. All of these amounts fall between a few dollars and the high twenties per data room. The real cost is time. A single run can take an hour or more, which makes this kind of work a background job that is handed off and collected later, not an interactive query.

What vertical AI can still defend

With the harness public in method and the model still deciding much of the score, the last sentence of the original call is the one left exposed. That sentence put the defensible position for vertical AI in a task-specific harness plus a post-trained open-weight orchestrator, rather than in reselling frontier models or wrapping a general agent. Three points take that conclusion apart:

1. The harness itself is public. The Recursive Language Models inference library is open source under the MIT license, and the method has already been packaged as a ready-made module in general agent frameworks. Harvey's post also describes the approach in considerable detail.

2. Post-trained open-weight models currently lose to frontier models. The post-trained Qwen root scored 63.0%. That is 14.6 points below Opus 5 in the same harness, and below Grok 4.5, GPT-5.6 Sol and GLM-5.2. None of those had post-training. An open-weight root may still win on cost and deployment control, since open weights can be self-hosted. Harvey has not published the per-room cost of that model, so that advantage cannot be stated yet.

3. Harvey's own post-training used no customer data, and the result is not in the product yet. Tenet, the in-house model Harvey announced in August, is a Kimi K3 base post-trained together with Fireworks. The post states: "We did not use any customer data in any of our post-training efforts." Harvey says the resulting capabilities will be brought into the Harvey product "over time".

From this data, the behavior that lifts the score has to be trained, and that training needs its own environment. The environment needs simulated data rooms, expert checklists of five hundred or more items, and the judge's checklist scores fed back as a reward, round after round. Our read: the scarce asset may be the yardstick and the training environment built around it, rather than the code that reads the documents. The yardstick here is the grading standard that judges whether a diligence review is complete and correct. This reading has two limits. Part of Harvey's legal agent benchmark is already open source, and how much of that yardstick remains proprietary is unresolved. Channel advantages, such as client trust and contractual requirements that data never leaves the firm, are not measured in this piece. Everything in this paragraph is inference. This piece holds it with less confidence than the four sections before it.

What this does to this publication's trend tracking: the tracked question is who ends up owning the general-agent layer. There are two paths. Model vendors can build it themselves as a general agent platform with a swappable underlying model, such as Claude Code. Or a vendor-neutral third party can build an outer layer that switches between models, such as Databricks' Omnigent. The evidence here lends some support to the first path. On long-document work, the gap between a tailored harness and a general agent comes mainly from whether the work is farmed out and read to the end. Model vendors are putting exactly that capability into their own general agents, as with Claude Code's dynamic workflows. If the tailored harness's edge disappears on general agents, vertical AI companies serving a single industry such as law or finance have less to defend at this layer. Our read, held with low confidence: what remains is the grading standard and the channel.

Our call

On long-document work that exceeds a model's reading limit, quality tracks whether the agent reads the material to the end. In Harvey's experiment, agents in the recursive harness read more than a tenth of the data room in nearly every run and most of it in most runs, while agents in the simple tool loop read under 1%. Across the same seven models, the two harnesses' average pass rates were 39 points apart. Harvey reports coverage and score as moving together and has not separated cause from correlation. Finishing the reading depends on the habit of delegating the work. Harvey's experiment shows that habit needs no labeled legal knowledge and can be learned from a small number of traces. Harvey has verified that learnability on one open-weight model only, GLM-5.2. Anthropic has also already put the ability to spawn hundreds of sub-agents at once into a general agent, Claude Code. A general agent run on defaults reads too little; with delegation switched on, it does not necessarily lose. Once the harness is right, the choice of root model still moves Harvey's pass rate by more than 35 points, and the post-trained open-weight root still trails frontier models such as Claude Opus 5. So what a vertical AI company can defend is not the harness code, and for now not a model it post-trained itself. Our call, held with low confidence: the defensible asset is the expert grading standard that judges whether a diligence review is complete and correct, together with the simulated data rooms that let a model be trained against it repeatedly.

What to watch next

1. Whether anyone reruns a general agent with sub-agent delegation switched on, on long-document diligence. This is the most direct test of this piece. Places to watch: follow-up posts on Harvey's blog, and updates to Anthropic's dynamic workflows.

2. Whether Harvey's post-trained root agent reaches the product, and at what cost per data room. If a post-trained open-weight root gets near-frontier scores at a clearly lower cost, point two under "What vertical AI can still defend" has to be rewritten.

3. Whether the diligence task set of the legal agent benchmark is published, and whether the agreement between the model judge and lawyers is disclosed. Once the yardstick is public, the claim that the yardstick is scarce has to be withdrawn.

What would prove this wrong

The verdict date for all three conditions is March 31, 2027. The thresholds, twenty points and 77.6%, are observation lines set by this publication.

1. On the same kind of long-document diligence task, Claude Code or Codex with sub-agent delegation (or dynamic workflows) switched on still trails the task-specific recursive harness by twenty points or more. Then this piece's claim that the gap is mainly unfinished reading fails, and the original call's sentence 3, that general agents are the wrong default, is restored.

2. A post-trained open-weight root agent in the same harness matches or beats the best frontier model. One example would be GLM-5.3, which Harvey is now training, scoring no lower than Opus 5's 77.6% on the held-out set. Then the claim under "Harness beats model: half right" and "What vertical AI can still defend" that post-trained open-weight models currently lose to frontier models has to change, and the second half of the original call is partly restored.

3. A third party redoes the test with real data rooms or lawyer grading, and the relationship between coverage and score disappears. Then this piece's mechanism claim fails.

Written from the same research and judgments as the Traditional Chinese edition. Sources are linked; original documents are kept distinct from reporting about them, and unverified points are marked as such.

Sources cited in other language editions

These sources are linked in the body of this piece's other language editions but not in this one.

• latent.space/p/rlm
• magazine.sebastianraschka.com/p/using-local-coding-agents
• arxiv.org/html/2605.27922v1
• arxiv.org/abs/2606.12344