预测奖励会诱导大模型陷入虚假成功陷阱，通道比内容更重要。
The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works
- 用预测误差做每步奖励会导致优化器构造出虚假的‘黑暗房间’陷阱
- 无论模型大小或任务，只要不调整标准差归一化就必崩溃，准确率100%但任务失败
- 奖励信号是否危险取决于其组内方差变化轨迹，而非内容本身
密集的每步监督是解决稀疏奖励长时序大模型智能体的标准方法：通过奖励策略对下一步观测的预测来实现，理论上在基于潜在函数的塑造下是安全的。已有研究报道预测奖励和辅助损失变体存在成功与不稳定并存的现象；本文提供受控分析：74个预注册实验在ALFWorld、WebShop、合成部分可观测马尔可夫决策过程（POMDP）以及Qwen3-1.7B/4B/8B上，固定预测信号，仅改变传递机制，使用GRPO训练。结果表明：(1) 所有未修改标准差归一化的差值形式奖励运行均崩溃（11次实验跨规模、系数、分组），其中合成环境因下限约束停止；ALFWorld中预测准确率达1.0但成功率降为0，优化器构建了‘黑暗房间’。数学推导仅一行：全失败组的z-score归一化抵消了塑造系数；仅移除标准差归一化即可恢复基线性能。(2) 信号危险性由组内方差轨迹决定，第二维度为可劫持性，该指标可回溯所有奖励通道崩溃，并通过预注册前瞻性测试验证。(3) 同一信号作为教师强制的辅助损失在4B模型上无害，但增益并非来自信号本身：内容无关的安慰剂类表现匹配甚至超越‘黄金’信号（种子s0: 78.8 vs 68.6；s42: 67.9 vs 57.9），辅助更新才是正则化关键。(4) 在8B模型上，配方变为双稳态：全权重黄金信号锁定三种子中的两个；所有内容无关或权重降低的实验均保持健康。在ALFWorld和WebShop上，任何奖励通道变体均无法显著优于匹配归一化基线，且黄金信号也无法显著超越内容无关安慰剂：决定成败的是通道而非内容，安全通道依赖于具体训练范式。
原文摘要 · Abstract (English)
Dense per-step supervision is the standard remedy for sparse-reward long-horizon LLM agents: reward the policy for predicting its next observation, which looks provably safe under potential-based shaping. Published prediction-reward and auxiliary-loss variants report both successes and instabilities; we supply the controlled account: 74 preregistered arms dissect one fixed prediction signal under GRPO across ALFWorld, WebShop, a synthetic POMDP, and Qwen3-1.7B/4B/8B, varying only the delivery mechanism. (1) Every run sustaining this difference-form reward under untouched std normalization (no filtering, dynamic-sampling, or decoupling mitigations) collapses: eleven runs across scales, coefficients, group sizes, and groupings (the floor-bound synthetic environment stalls instead); ALFWorld runs end in an absorbing state (prediction accuracy -> 1.0, success -> 0): the optimizer builds the "dark room". The algebra is one line: in all-fail groups z-scoring cancels the shaping coefficient; removing only std normalization restores baseline parity. (2) A signal's danger is set by its within-group variance trajectory, plus hackability as a second axis; it retrodicts every reward-channel collapse and survives preregistered prospective tests. (3) The same signal as a teacher-forced auxiliary loss is harmless on ALFWorld at 4B, but the gain is not the signal's: content-free placebos as a class match or beat gold at both matched seeds (s0: 78.8 vs 68.6; s42: 67.9 vs 57.9); the auxiliary update is the regularizer. (4) At 8B the recipe turns bistable: gold full-weight locks two of three seeds; every content-free or reduced-weight arm stays healthy. No ALFWorld or WebShop reward-channel variant measurably beats its matched-normalization baseline and no gold signal measurably outperforms its content-free placebo: the delivery channel, not the content, decides; which channel is safe is regime-dependent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。