SecondSourceAI 產業的第二來源

分析 · 根據兩個以上獨立來源 · 寫於 2026-11-15

在獨立團隊自建的評測裡,需求沒明列、要 agent 從文件或程式環境自行推得時,受測 agent 的成功率都明顯低於需求明列時(Word 文件格式評測最佳系統 62.9%→40.7%;軟體工程與研究任務八個模型掉 53–67 個百分點);另一個多類工作評測在目標規格不全時,完成率也低 57.2 點。真實開發者與 coding agent 的工作階段也顯示,初始指令漏掉後來才冒出的必要需求很常見(抽樣軌跡 56%),而且這是 agent 誤讀意圖、被開發者糾正的一類原因。所以只看明確指令下的通過率,會高估 agent 在真實工作中的可用度;評估與產品驗收要加入規格不全或須自行推斷的變體。保留兩個矛盾:明確指令本身也常失敗(真實工作階段裡最大的出錯原因是指令遵循失敗;評測裡需求全部明列時,最佳系統也只有 62.9%);事前主動向使用者提問的架構能補回大部分落差,所以缺口未必在「沉默推斷」本身,也可能在 agent 不會察覺該問。

這不是預測,所以沒有排定哪一天對答案。

什麼會推翻它

各條行尾若有日期,那是相反的證據要在那天前出現才算數的期限,不是對答案的日子。

論文作者或第三方在 2026 年 9 月之後上市的旗艦 agent 上,用同一批任務公布「需求明列」與「需求須從文件或程式環境自行推得」兩種條件的通過率(文件格式編輯評測或軟體工程任務評測的新版、重跑皆可) — 至少兩個獨立評測中,最佳系統兩種條件的通過率差距都縮到 5 個百分點以內 — 差距縮到門檻內,即推翻「只看明確指令下的通過率會高估真實可用度」 — 2027-06-30

公開的真實使用研究:分析至少一千個開發者與 coding agent 工作階段的指令完整度或出錯原因 — 初始指令漏掉必要需求的比例低於 15%,或「指令沒講明」占可判定出錯原因低於 5% — 低於門檻,即削弱「真實工作中的需求常沒講明、且是 agent 出錯原因之一」這條前提 — 2027-06-30

寫下判斷時的出處

WordInferBench: Can Agents Infer How to Edit from Word Document Context? · arXiv (Ding/Huang/… Si-Qing Chen/Lei Cui/Furu Wei — Microsoft Research 系;含 Nanning Zheng, 西安交大)
「the best-performing evaluated system, Claude Code with Opus 4.6, achieves a 62.9% task-level pass rate on descriptive variants but only 40.7% on inferential variants, indicating that current systems complete tasks less reliably when the required formatting changes must be determined from document context.」

https://arxiv.org/html/2611.15189v1
「All evaluated agent systems obtain lower pass rates on inferential variants than on descriptive and referential variants.」
「Separately, in the descriptive condition, both the properties and their target values are explicitly specified, yet the same agent achieves a pass rate of only 55.4%.」

https://arxiv.org/abs/2610.17757
「Evaluating eight frontier models including GPT-5.6-Sol, we find outcome alignment broadly missing: the models lose 53 to 67 points of success rate when moving from the full specification to the partial instruction, and consequence feedback across four submissions does not close the gap.」
「In every task the user genuinely lacks part of the requirements, each requirement stays discoverable from the code and runtime behaviour of the environment, and requirements the agent infers are separated from those the user adds after observing delivered work.」

https://arxiv.org/html/2610.18090
「Goal underspecification has the lowest Completion, 57.2 pp below Full.」
「Figure 3 shows that Full performance does not ensure an advantage under underspecification.」

https://arxiv.org/html/2610.17757v1
「We find that 56% of the sampled trajectories have incomplete initial instructions, indicating that the initial request does not cover all requirements that ultimately emerge during task completion.」

https://arxiv.org/html/2605.29442v2
「In 44.10% of these episodes, the developer leaves a consequential gap in the prompt, such as agent behavior or implementation scope (C1. Underspecified Instruction).」
「S2 reflects not wholesale misunderstanding, but a plausible concretization of an underspecified request.」
「Instruction-Following Failure (C6, 36.49%) is the largest cause, with 94.18% of attributions supported by direct log evidence」

https://arxiv.org/abs/2603.26233
「Across both proprietary and open-weight frontier LLMs, our scaffold achieves a 69.40% task resolve rate, significantly outperforming a standard single-agent setup and closing the performance gap with agents operating on fully specified instructions.」

https://arxiv.org/abs/2502.13069
「However, when models interact for underspecified inputs, they effectively obtain vital information from the user leading to significant improvements in performance, up to 74% over the non-interactive settings, underscoring the value of effective interaction.」