Trend · preprint, October 23

Harness rank shifts by model, 52,168 runs; no scores shown

A preprint by Shao Zhang, Hangfan Zhang and co-authors tested 4 general-purpose coding-agent harnesses with 4 models on 6 benchmarks, 52,168 runs in all, and found that harness rankings change with the model and benchmark underneath. The harnesses tested were OpenCode, Codex CLI, Pi CLI and DeepAgents (arXiv 2610.32483, 2026-10-23). According to the abstract, OpenCode did best across models, but no harness was best on every model and benchmark. Pairings with high task scores didn't always have high valid-completion rates. The valid-completion rate is a metric the authors defined: whether the agent ran to the end normally and handed back a result that could be scored. Of the 4 models, the abstract names only DeepSeek V4 Flash Preview, and it lists no scores. Another October 23 preprint, from a Singapore Management University team, ran the same open-source model in six harnesses on SWE-bench Verified and found solve rates differing by up to 44 percentage points (arXiv 2610.32131, 2026-10-23). The Zhang paper tests the same point across 52,168 runs. Our reading: a harness ranking on its own may flip when the model changes, so agent leaderboards need to state which model and which harness were paired. How we checked →

Concept illustration of a hand on the right holding a tablet whose screen shows a black header bar lettered PREPRINT above two side-by-side columns of four stacked window cards, each with three small dots and grey lines. The left column runs, from top, a red card lettered OPENCODE, then black, beige and grey-brown cards; the right column runs black, a red card lettered OPENCODE, grey-brown and beige. Below each column is a laptop, black on the left and grey on the right, each screen showing a speech bubble filled with grey lines, with a pale pink circle in the background at upper right.

Source and checks

This item ran in the brief of Oct 27, 2026, alongside 12 other items. How we checked it, the quote checks and what was still missing that day are on that issue's How we checked page.

Cite this item

SecondSource (2026). "Harness rank shifts by model, 52,168 runs; no scores shown." https://secondsource.io/en/research/2026-10-27-1/. Accessed access date.