A harness evaluation published by Y Combinator CEO Garry Tan found that swapping the harness around the same model left 20-task pass rates indistinguishable, while cost per solved task differed by more than twofold.

Concept illustration: a black-framed display on a stand, tilted with its right side higher, has a black band with white capital lettering across the top of its screen. Below the band a thin black line runs across the screen and holds, from left to right, a card lettered QM containing a speech bubble with three grey bars, a short stack of beige discs, a card lettered OPENCLAW containing the same kind of bubble, and a much taller stack of red discs that rises to near the black band. Printed on the image: About 2.3x Cost-per-solve gap from Tan's self-reported data.

On this public task set the differences worth comparing are cost and time, according to Garry Tan's self-reported development results, and Tan and YC both have stakes in some of the harnesses. On Opus 5.5, YC's harness QM passed 72.5% at US$0.19 per solve, Claude Code 76.3% at US$0.22, Hermes Agent 78.8% at US$0.24 and OpenClaw 76.3% at US$0.44. The most expensive costs about 2.3 times the cheapest (our calculation). Time per run was 40.1 seconds for both QM and Claude Code, against 99.4 seconds for OpenClaw. The README states that pass rates across the pairings can't be told apart, and that these are development results on public tasks, not a held-out confirmation test. The README also discloses YC's investment in Hermes (g-harness-evals). How we checked →

Source and checks

This item ran in the brief of Nov 9, 2026, alongside 14 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.

Cite this item

SecondSource (2026). "A harness evaluation published by Y Combinator CEO Garry Tan found that swapping the harness around the same model left 20-task pass rates indistinguishable, while cost per solved task differed by more than twofold.." https://secondsource.io/en/research/2026-11-09-1/. Accessed access date.