Trend · preprint, October 23

SMU: model's SWE-bench gap hits 44 pts; no frontier test

A preprint by three Singapore Management University researchers ran the same open-source model in six harnesses on SWE-bench Verified, and the solve rate varied by up to 44 percentage points. The authors are Lu Zhao, Christopher M. Poskitt and David Lo. A harness is the software around a model that gives it tools and runs its loop through a task. The study used 6 open-source harnesses, 3 lightweight open-source coding models and 100 SWE-bench Verified tasks. It ran 1,800 times under the same serving, billing and scoring conditions, once per combination (arXiv 2610.32131, 2026-10-23). Switching the model reshuffled the harness ranking. Among harnesses that let the model choose its own next step, the cheapest and the most expensive differed by more than 20 times in billed tokens per solved task. These results are the authors' own, the readings come from the abstract, and no frontier closed model was tested. Our read: a leaderboard score measures a pairing of model and harness, and can't be credited to the model alone.

Concept illustration of an open magazine spread on a beige backdrop, topped by a black banner lettered in white "SINGAPORE MANAGEMENT UNIVERSITY", with a laptop on each page whose screen shows a red-outlined window labeled "HARNESS" and a speech bubble (the left window also has a sidebar list, the right one a text panel along the bottom). In front of each laptop lies a sheet headed "SWE-BENCH VERIFIED" with grey text lines, and a striped wastebasket stands at each outer edge; a black insect sits in the left wastebasket and another on the right-hand sheet, with a pale pink circle in the background at top right.

Source

This item ran in the brief of Oct 26, 2026, alongside 5 other items.

Cite this item

SecondSource (2026). "SMU: model's SWE-bench gap hits 44 pts; no frontier test." https://secondsource.io/en/research/2026-10-26-1/. Accessed access date.