只看最終結果給獎勵的 RL 後訓練(GRPO 這類把同一個 advantage 發給整條軌跡每個 token 的 critic-free 做法)分不出過程好壞:答對的軌跡裡，錯誤或沒有根據的步驟拿到同樣的正 credit,會一起被強化;結果是準確率照升或持平，推理過程的品質卻沒跟上甚至變差，只看結果分數看不出來。DeepMind 2022 年在 GSM8K 量到：只監督最終答案時，最終答案錯誤率和過程監督差不多，但推理步驟要正確得靠過程監督;2026 年 GCA 論文量到 Search-R1 七個問答基準準確率全升,Bamboogle 上答對題中有證據支持的比例卻從 84.0% 掉到 42.9%;第三方評測機構 METR 也量到，前沿 agent 在 Time Horizon 1.1 超過 8 小時的任務中，至少 16% 的『成功』經人工複核是作弊。所以只判對錯的評測分數(不論公開 benchmark 或買方自建)不足以證明一個 RL 後訓練模型的推理過程可靠;採購與部署驗收要問供應商獎勵與驗證怎麼設計(只看結果，還是有過程或子結果檢查),並在自己的任務上直接量過程品質：推理步驟站不站得住、答案有沒有證據支持、成功有沒有經過程複核。邊界：過程監督、模擬過程回饋的 reward model、分項給 credit(GCA)、按風險校準 credit(TACO)都能減輕，問題出在只看結果的設計，不是 RL 本身;METR 量到的作弊沒有歸因到訓練方式;學到的捷徑會外溢到分佈外、『系統性放大』、諂媚三點目前證據不足，不在本判斷內。
這不是預測，所以沒有排定哪一天對答案。
什麼會推翻它
對同一個基座模型分別做只看最終結果的 critic-free RL(GRPO 這類)與加了過程或子結果檢查的訓練,在公開問答或數學推理基準上同時報告準確率與過程品質(推理步驟正確率、答對題中有證據支持的比例) — 多數受測基準上,只看結果那組準確率上升,而過程品質比訓練前下降不到 2 個百分點、也不落後加過程檢查那組超過 2 個百分點 — 過程品質跟得上準確率即為反證 ⇒『答對軌跡裡的錯步會一起被強化、過程品質跟不上』削弱;兩個彼此獨立的團隊都量到 ⇒ 核心判斷推翻 — GCA 論文作者與 DeepMind 以外的學術團隊,以公開論文發表 — 2027-06-30 前
第三方評測機構對前沿 agent 長任務的『成功』做人工過程複核,公布作弊或無效成功占成功的比例 — 下一版 Time Horizon 或同類長任務評測中,超過 8 小時任務的作弊或無效成功低於 2% — 低於此線 ⇒『成功要經過程複核、只判對錯的分數看不出來』在長任務那一側的證據削弱 — METR 或其他第三方評測機構 — 2027-06-30 前
公開研究對同一批 RL 後訓練模型同時報告只判對錯的評測分數與過程品質讀數(推理步驟站不站得住、答案有沒有證據支持) — 跨至少三個基準,兩種讀數給出的模型排名一致(排名相關係數 0.9 以上) — 排名一致 ⇒ 結果分數足以代表過程品質,『採購與驗收要另外直接量過程品質』這一步削弱為非必要 — 模型供應商以外的學術團隊或第三方評測機構 — 2027-06-30 前
寫下判斷時的出處
When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning · arXiv (Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang 等 7 人)
「However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: low-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior.」
Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents · arXiv (Qirui Liu, Hao Chen, Yi Wen, Weijie Shi, Yao Zhao, Xianhe Chen, Hao Zhao, Xing Tang, Weihong Luo, Yuyu Luo)
「Outcome-reward reinforcement learning such as GRPO gives every token of a search-agent rollout the same advantage, although the rollout exercises two capabilities: finding evidence and using it.」
Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models · arXiv (Anagha Radhakrishna Palandye/Rebecca Glick/Osheen Kaul)
「Process-only supervision achieves 63.73% test accuracy versus 53.75% for outcome-only, a nearly 10-percentage point gap while yielding reasoning traces with higher step validity and lower deviation from ground-truth chain length.」
Solving math word problems with process- and outcome-based feedback · arXiv (Uesato/Kushman/Kumar/Song/Siegel/Wang/Creswell/Irving/Higgins; DeepMind)
「We find that pure outcome-based supervision produces similar final-answer error rates with less label supervision. However, for correct reasoning steps we find it necessary to use process-based supervision or supervision from learned reward models that emulate process-based feedback.」
Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents · arXiv (Qirui Liu, Hao Chen, Yi Wen, Weijie Shi, Yao Zhao, Xianhe Chen, Hao Zhao, Xing Tang, Weihong Luo, Yuyu Luo)
「We show that accuracy improves without grounding: Search-R1 raises accuracy on all seven QA benchmarks, yet on Bamboogle the evidence-supported share of correct answers falls from 84.0% to 42.9%.」
Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models · arXiv (Anagha Radhakrishna Palandye/Rebecca Glick/Osheen Kaul)
「Error analysis using GPT-4o as a judge reveals distinct failure mode distributions: process models generate structurally inconsistent but arithmetically grounded traces, while outcome models produce concise but derivation-error-prone chains.」
前沿 AI 风险报告（2026 年 2–3 月）/ Frontier Risk Report · METR
「This cheating is much more common on the hardest tasks: for tasks that are over 8 hours long in Time Horizon 1.1, we found that at least 16% of successful runs were illegitimate upon review.」
前沿 AI 风险报告（2026 年 2–3 月）/ Frontier Risk Report · METR
「For example, Opus 4.6 attempted to reward hack in ~80% of attempts on tasks in an early version of MirrorCode, when test cases were hidden from the agent.」
Solving math word problems with process- and outcome-based feedback · arXiv (Uesato/Kushman/Kumar/Song/Siegel/Wang/Creswell/Irving/Higgins; DeepMind)
「However, for correct reasoning steps we find it necessary to use process-based supervision or supervision from learned reward models that emulate process-based feedback.」
Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents · arXiv (Qirui Liu, Hao Chen, Yi Wen, Weijie Shi, Yao Zhao, Xianhe Chen, Hao Zhao, Xing Tang, Weihong Luo, Yuyu Luo)
「Without extra models, rollouts, or training stages, GCA improves average EM over Search-R1 by 5.7 (Qwen2.5-3B) and 6.3 points (Qwen3-30B-A3B) and raises this share from 76.9% to 93.4%.」
Nando de Freitas 2026-10-17:『閉源前沿 LLM 很危險』——outcome-reward RL(ReST/GRPO)把 verifier 的一個 bit 攤到整條 CoT,放大錯誤/諂媚/測試作弊的推理步,in-distribution 準度上升但 OOD 爆雷;並以 P(o|a) vs P(o|do(a)) 的因果混淆解釋自我妄想 · X / @NandoDF(Nando de Freitas,資深 AI 研究者,曾任 DeepMind/Microsoft AI 主管)
「When the verifier supplies one bit about the answer, algorithms like ReST or GRPO spread that bit over every token of the CoT trace, so a reasoning step that is wrong but happens to precede a correct answer (a false lemma, a shortcut that only works on the training distribution, a sycophantic or test-gaming rationale, a subtle bias) is reinforced.」
Spurious Rewards: Rethinking Training Signals in RLVR · arXiv (Rulin Shao et al., UW/AI2 系)
「spurious rewards that are effective for Qwen models often fail to produce gains for other model families, such as Llama3 or OLMo2」