Newer Claude models invent fields on third-party coding tools, Armin Ronacher reports. The cause is traced and one switch fixes it. Our call: buyers should measure format failures in their own tool layer, not leaderboard gaps

Key takeaways

In late August a Chinese-language relay went around the usual channels. The relay said that Pi, a coding agent that runs models from several vendors through its own tool layer, had written in its development notes that the newest Claude models, when connected to a tool layer other than Claude Code, generate parameters the tool never defined, with names like `requireUnique`, `matchCase` and `oldText2`. On the same day, two evaluators ran the same anonymous model on the same 113-task suite and got 58.4% and about 63%, and both attributed the gap to running different tool layers. Last night this publication filed an internal call that merged those two items: model and tool layer cannot be separated. The call did not run in the daily brief. Its reasoning was that the measurement is noisy and that moving a model to another tool layer really does make it worse. The proposed mechanism was post-training that fits the model too tightly to the vendor's own tool definitions.

This piece takes that call apart and re-tests it. The result has three layers. First, the sourcing was wrong. Those three parameter names are not news from August 22. The names come from a July 4 blog post by Armin Ronacher, one of Pi's authors, which this publication had on file in July and which last night's call never cited. The other set of numbers, savings figures from 19 sessions, comes from a pull request an outside contributor opened on August 15. A bot closed it automatically because the author was not an approved contributor, and it was never merged. Second, the mechanism is more specific than last night's version, and more fixable. The fields the model invents use the other tool's naming style but follow the rules of Claude Code's own edit tool. One such rule is that the text being replaced must occur exactly once. The author who made the observation does not explain this as the model memorising its home field names. Ronacher's explanation is that the home tool layer is too forgiving: extra fields are silently dropped, so during training the model was never penalised for writing them. The direct consequence of that reading is that the fix sits on the tool layer's side, as a strict-mode switch or one line of code that filters unknown fields. Strict mode forces the model's output to match the tool definition. Third, the 4.6-percentage-point gap says nothing about the models, but the log of one of the two runs holds a count that does. On 113 tasks the sampling error of a single run is itself about 4.6 percentage points, so the gap between the two runs sits inside the noise. The public log of the 58.4% run also reports that 11 of 113 attempts, 9.7%, were aborted because the model returned three consecutive responses with no tool call in them. That abort count measures how well a model fits a tool layer. It is a different measure from the pass-rate gap, though at 9.7% it is about twice the size of the 4.6-point gap. It comes from a single run, and there is no matching count from another tool layer to compare it with.

If only one paragraph gets read, make it this one: the question for a tool-layer vendor now has a standard form. Which tool definition does the vendor use for each model, is strict mode on by default, and how long after a new model ships is that row filled in.

The sections below fix the sourcing, explain the mechanism, set out what the two vendors document, swap the 4.6-point gap for the abort count, test the strongest rebuttal, and describe the unit of procurement.

One result would overturn this. If anyone turns strict mode on over Pi's original tool definitions and still measures an edit success rate clearly behind Claude Code, the narrowing to "fixable" is wrong and the stronger version of last night's call comes back.

The tree is this publication's map of the coding-agent line. This piece works on the right-hand branch, the claim that tool layer and model cannot be separated. In the tree, "harness" means what this piece calls the tool layer. That is the software around the model that defines and runs its tools. For the first time that claim has a sourced mechanism, and the mechanism comes with a fix.

One: fix the sourcing. Not two sources colliding on August 22, but one July blog post and one unmerged pull request

Last night's call rested on a skeleton of "same day, two unrelated sources". For that skeleton to stand, both sources have to be independent and both have to be new. Checked, neither condition holds.

With the sourcing fixed, three things change. First, the mechanism half of the story moves up from a secondhand Chinese relay to a first-hand blog post plus a GitHub issue, and that issue #6278 was opened on July 3, a day before the blog post. Second, the numbers half moves down from "vendor self-report" to "an example extension from an outside contributor that the vendor did not accept". Those figures can no longer be presented as Pi's claim, and still less as any vendor's measurement. Third, on August 23 a Taiwanese writer working through the same thread had already written that the three field names could not be found in Pi's merged documentation. That writer's search was sound. The names are not in Pi's documentation. They are in Ronacher's blog post.

Evidence shaped like "two sources collided on the same day" gets its dates checked first, every time.

Two: what "inventing parameters" actually is. Tool definitions, two vendors' edit tools, and a training ground that forgives too much

Three terms first.

Tool definition. Before a model can call a tool, a JSON description tells the model what the tool is called, which parameters it takes and what type each one has. The tool call the model returns has to match that description. Otherwise the tool layer either rejects the call or guesses.

Claude Code's edit tool. According to Anthropic's tools reference, the tool is called Edit and takes four parameters: `file_path`, `old_string`, `new_string`, and an optional `replace_all`. The rule is that `old_string` must occur exactly once in the file. If it occurs more than once, the model has to supply more surrounding context or set `replace_all` to true.

Pi's edit tool. According to Pi's release notes from late March, the tool takes a `path` and an `edits` array, and each entry in the array has an `oldText` and a `newText`. The names are camelCase and several edits can be sent in one call.

Now the fields the model added. Ronacher lists them as `type, id, kind, unique, requireUnique, matchCase, in_file, forceMatchCount, children, notes, cost, oldText2, newText2, oldText_2, newText_2`, plus one `event.0.additionalProperties`. Held against the two definitions above, the shape is clear:

• The naming is Pi's. `oldText2` and `newText_2` are Pi's own `oldText` and `newText` with a number appended, not Claude Code's `old_string`. The model did not carry its home field names across.
• Claude Code supplies the rules. `unique`, `requireUnique` and `forceMatchCount` all express "this text must occur exactly so many times". That is the central rule of Claude Code's edit tool, and Pi's definition has no such concept.

Put together, what the model carries over is not the literal text of its home tool but its home tool's habits: several edits per call, and a way to say "this span must be unique". The model wrote those habits out in the other tool's naming style. That is narrower than last night's "over-fitted to its own tool definitions", and easier to test.

Ronacher's own explanation goes one level further down. In his words, Claude Code's edit tool "silently filters out unexpected keys and it does not use `strict` mode either", so extra fields are dropped without a trace. He goes on: "If reinforcement learning happens in a harness like that, or a simulation of one, then slightly malformed tool calls can still complete the task and receive reward". His conclusion: "The harness fully absorbs the error and there is little gradient against inventing an alias, adding a stray field or using a nearby parameter name".

That explanation has three checkable predictions, and all three line up. First, it predicts that older models lack the fault and newer ones have it, because the fault is a side effect of post-training that tracks the home tool layer ever more closely. Ronacher observes that "both Opus 4.8 and Sonnet 5 show it but none of the older models". Second, it predicts that the fault disappears once the format is enforced. Ronacher writes: "Obviously one could turn on `strict` sampling in Anthropic and the problem should go away". Third, it predicts that the fault is invisible inside the home tool layer, which drops extra fields by design. Nobody has measured that, but it explains why the behaviour was found by the author of a third-party tool layer and not by Claude Code's users.

The only frequency figure comes from the issue. In one user's session, about 20% of the responses for one particular edit call were malformed this way. That is one session, one user, one model, not a population figure. Nothing later in this piece multiplies by it.

Three: what the vendors say. One vendor writes "trained on this tool" into the documentation with a number; the other vendor's documentation has no such sentence

Last night's call treated "the model was trained on its own tool definitions" as Pi's guess. One vendor has in fact written it into its own documentation.

OpenAI's GPT-5.1 prompting guide states: "GPT-5.1 has been post-trained on specific tools that are commonly used in coding use cases". It names `apply_patch` and `shell`. The same document attaches a number: "In testing, the named function decreased apply_patch failure rates by 35%". The named function is the tool type that OpenAI defines, used in place of a self-written function with the same job. The official API documentation lists the tool as supported from GPT-5.1 through GPT-5.5.

This matters because it turns last night's "guess" into a vendor's own statement, with a measured gap between the trained tool shape and an untrained one. It also explains another of Ronacher's observations: "So far, the Codex models I tested did not show this type of regression". Our read: OpenAI shipped the trained tool shape as a type anyone can use on the API, so a third-party tool layer that connects to it is running exactly the shape the model was trained on. Anthropic's trained edit shape lives inside Claude Code, and a third-party tool layer does not use it by default.

On Anthropic's side, this piece found two documents that each say half of the story. The text editor tool documentation offers an Anthropic-defined `str_replace_based_edit_tool` in three versions, dated from January to July 2025. The earliest version's note says it "is optimized for Claude Sonnet 3.7". That document carries no sentence saying the model was trained on the tool, and that has to be recorded as it is. The second document covers strict tool use. With a tool definition marked `strict: true`, the model's output is sampled under a grammar constraint and is guaranteed to match the definition. The supported models include Opus 4.8, Sonnet 5, Opus 5, Fable 5 and Fable 5.1. The price is that the definition must set `additionalProperties` to false and cannot use recursion, numeric ranges and a few other schema features, and the first use of a given definition waits for the grammar to compile, after which it is cached for 24 hours.

So the switch Ronacher says should make the problem disappear exists on Anthropic's API, and both models he reports the fault on are on the supported list. The Pi issue points down the same road: "Anthropic API offers strict schema validation that ensures this".

Four: drop the 4.6 percentage points and put the 11 aborts from the same record in their place

The numbers half of last night's call was this: same model, same suite, two evaluators at 58.4% and about 63%, both attributing the gap to the tool layer. The problem with that attribution is not motive. It is arithmetic.

The suite has 113 tasks. 58.4% is 66 passes. At that pass rate the standard error of a single run over 113 tasks is about 4.6 percentage points (our calculation). The error on the difference between two independent runs is about 6.6 percentage points. A 4.6-point gap between the two evaluators is 0.7 of one error unit. That gap says nothing about the cause, whether it is the tool layer, temperature or luck. The two evaluators pointing at the tool layer is an honest guess, not a measurement.

But the same record holds a number that can be used. The 58.4% run comes with a public per-task log. The log names the tool layer as `pier` 0.3.1 together with mini-swe-agent, the agent scaffold the run used, and it states that 11 of 113 attempts, 9.7%, were aborted because the model returned three consecutive responses with no tool call in them. The log attributes these failures to "tool-call format errors rather than reasoning deficits".

Those 11 aborts measure whether model and tool layer fit. They belong to the same family as the invented fields in section two: the model was supposed to produce something the tool layer could read, and produced something it could not. At 9.7% of attempts it is about twice the 4.6-point gap, though the two are different measures. It comes from a single run, so no comparison across evaluators is needed to get it. The log has no matching count from another tool layer, so this is an absolute count, not a gap.

Set against a larger population, the shape is the same. Harness-Bench, a benchmark from Peking University and Qiyuan Tech that runs the same tasks through six configurable tool layers and eight model backends, recorded 5,194 execution trajectories. On the same tasks and the same pool of models, the best and worst tool layer were 23.8 points apart. In the paper's own failure taxonomy the largest class is contract and format violations, at 36.4% of what it defines as execution-alignment failures, followed by failure to recover after a tool error at 24.6%. This is a single self-reported paper that nobody has reproduced elsewhere, but it gives the shape this piece needs: of the gap a tool layer causes, the largest piece is format, not capability.

One more sample, already on file from August: the same DeepSeek V4-Flash was scored 82.7, 79 and 67 on TerminalBench 2.1 by three different tool layers, a spread of 15.7 percentage points. Those figures were compiled by one relayer, and the three original releases have not been checked individually. It shows how far scores spread across tool layers, but says nothing about whether the cause is format or capability.

Five: the strongest rebuttal. If one switch fixes the fault, why does the fault belong in a procurement discussion?

Now the attack on this piece itself. The strongest rebuttal runs like this:

Section two itself says the fix is `strict: true`, or one line in the tool layer that filters unknown fields. Claude Code already does exactly that. What does a bug that one line fixes have to do with "the unit of procurement moves from the model to model times tool layer"?

Half of this rebuttal holds, and that has to be conceded. For the failure shape in section two, extra fields, the answer really is that cheap: turn on strict mode, or do what Claude Code does and drop unknown fields. Ronacher also writes that the edit content itself is usually right; the content is correct and only the packaging fails. So last night's "moving a model over really does make it worse" narrows to: moving a model over hits a format regression that one switch repairs.

The half that does not hold has two parts.

The first part is the time lag. The switch is not something the model vendor flips for a customer. The tool-layer vendor decides it, per model. Ronacher's issue, opened July 3, points to a pull request that adds strict validation. The community grew two extensions at the same time. One is pi-anthropic-text-editor, which registers Anthropic's native `str_replace_based_edit_tool` for Claude inside Pi. The other is pi-apply-patch, which swaps Pi's write and edit tools for a Codex-style `apply_patch` when a GPT model is in use, and whose description says this is done to use the format the Codex models were trained on. Both extensions were split out of the same fork project. One has zero stars and the other sixteen. Both are tiny. What matters is the pattern: a table of "which tool shape for which model" is starting to appear inside tool layers, and third parties maintain it. The weeks between a new model shipping and that table gaining its row are the window in which "moving over makes it worse" actually happens. For the tool-layer vendor, the lag is a recurring maintenance cost rather than a one-off bug.

The second part is the layer this piece cannot measure. The switch solves "extra fields". It does not solve "different habits". Section two found that what the model carries across is rules and habits: several edits per call, a demand for uniqueness. Strict mode will stop the model writing those fields. It will not make the model give up the habits. The model may split the work into several calls instead, or pad `oldText` with longer context. Whether that layer costs success rate, nobody has measured. This is where the first condition under "What would prove this wrong" comes from.

One sample points the other way, and it has to go in, because it cuts against the simplification "use the vendor's own tool layer and the problem vanishes". Sebastian Raschka ran his own small test in late June. Rerunning his test cases, Raschka found, "to my surprise", that Qwen3.6 performed better through OpenAI's Codex tool layer than through its native Qwen-Code harness. Raschka himself discounts the result as a small sample. What it shows is that a model's home tool layer is not automatically its best one. What a model learns is the forgiveness and the habits of its training ground, not a brand name. A tool layer that is as forgiving as the training ground and compatible with its habits connects well even if it is not the vendor's own. So the table is not filled in with "whose tool layer". It is filled in with "for this model, which tool shape, and strict or not".

Six: so what does "model times tool layer" look like as a unit of procurement

Pulling the five sections together, last night's line "the unit of procurement moves from the model to model times tool layer" needs a change of unit. The right-hand side of the multiplication is a table with one row per model, rather than a vendor's brand:

The number to measure has changed. Model rankings do not carry across tool layers; this publication wrote that in July. What this piece adds is that the number worth measuring is the same model's tool-call format failure rate inside the buyer's own tool layer, like the 11 aborts in the 58.4% log. One run produces it, and no outside leaderboard is needed. Every row of Terminal-Bench 3.0, a leaderboard that scores a model together with its tool layer, already works that way. Its top row, Opus 5 with mini-swe-agent at 42.7%, is a score for that pairing. It is not a score for Opus 5 alone.

What to ask a tool-layer vendor for now has a standard form. It is the table above. A tool-layer vendor that claims multi-model support and cannot answer "which tool shape per model, what the strict-mode default is, how long until a new model gets its row" has multi-model support only at the interface. Nothing shows how the models behave inside it.

Open protocols and model portability are two different things, and this case makes the difference sharper. On August 23 Vercel declared that its agent tool fx would run only on three open protocols, MCP, Skills and Plugins, and the next day's release notes listed a fix to Codex and Grok sign-in as a selling point. The openness at the protocol layer is real, but it has no bearing on the table in this piece. Protocols decide which tools can be connected; the table decides whether the model invents fields once connected. One platform company can do both, but doing the first is not doing the second.

Our call

Models can be switched, but switching takes a per-model table rather than one line of configuration. For each model inside a given tool layer, the table records which tool shape is used, whether strict mode is on, and how long after a new model ships its row gets filled in. The invented-fields fault is fixed by one switch, so the fault is not proof that "moving a model over makes it worse". What the fault proves is that a tool layer has to maintain this table for every model, and the weeks when the table lags a new model generation are the window in which the regression actually happens. From here on, every claim that "model and tool layer cannot be separated" gets asked which row of this table it is about, rather than whether it backs "harnesses become commodities" or "harness and model are inseparable".

What this changes, and for whom

Companies signing annual contracts for coding models

"Switching models is one line of configuration" and "switching models means a full round of re-acceptance" are both blanket slogans. This piece replaces them with one question: has the tool layer filled in the row for the model under consideration? If the row is filled in, switching is close to a configuration change. If it is not, the first week can show a drop in success rate. The evidence so far suggests most of that drop is format rather than capability. That evidence is one run's abort count and one unreplicated paper, and the habits layer is unmeasured. Whether the threat of switching is credible at the negotiating table depends on how fast the tool-layer vendor fills in rows, not on the model vendor.

Companies building a tool layer or an agent platform

Claude Code's silent dropping of unknown fields now matters to tool-layer builders. That tool layer is the training ground for that vendor's models. The gap between its forgiveness and a third-party product's is the gap the model will show once connected to that product. The two cheapest moves follow from this piece's evidence. For models that support strict mode, strict is on by default. For models that have a native tool type, the native type is used rather than a rewritten function with the same job, and OpenAI's 35% failure-rate gap is the price tag on that line. Publishing the per-model tool-shape table is the one part of a multi-model claim that outsiders can verify.

What would prove this wrong

• Someone runs the same suite on the same model in Pi or any third-party tool layer, once with strict mode off and once with it on, and the edit success rate with strict mode on still trails Claude Code clearly. Then section five's narrowing to "one switch fixes it" is wrong, and last night's stronger version comes back: the regression lives in the habits layer, not the format layer. This is the part of the piece most likely to be overturned by a single result.
• Ronacher or anyone else publishes an invented-field rate for Opus 5 or Fable 5 on Pi's original definitions that is close to zero. Then "the newer generation is worse" is noise from one generation, and section two's generational direction is void. Ronacher says he did not test Fable, out of concern that a classifier might silently downgrade requests to Opus.
• Anthropic writes into its documentation that its models are trained on the text editor tool, and attaches, as OpenAI did, a failure-rate gap between the native type and a custom function. Then section three's asymmetry between the two vendors closes, and this piece's reading of the Anthropic side moves from "no such sentence found" to "both vendors in the same shape". The conclusion stands and gets harder.
• Any tool-layer vendor publishes per-model tool-call format failure rates, and the numbers show a negligible spread across models. Then the table in section six has nothing worth maintaining, and this piece drops to an engineering footnote about how to write tool definitions.

This piece sets its own observation window on the first two conditions, because that is where this set of questions is most likely to be answered next: the Pi issue thread and Ronacher's follow-ups. On the verdict date for this piece, November 12, 2026, this publication will check three things. Whether Pi has turned strict mode or the native tool on by default for Claude. Whether anyone has published an invented-field reading for Opus 5 or Fable 5. Whether any tool-layer vendor has published per-model format failure rates.

Written from the same research and judgments as the Traditional Chinese edition. Sources are linked; we distinguish original documents from reporting and mark what we could not verify.