用诊断迭代优化大模型生成的奖励函数，让稀疏结构任务的强化学习更有效。
When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL
- 将奖励设计视为调试过程，通过诊断和错误分类引导逐步改进。
- 在DoorKey-8x8任务中成功率从2.3%提升至97.6%，关键任务提升显著。
- 适合需要高可靠性接口的稀疏结构强化学习，尤其适用于复杂任务调试。
针对具有语义奖励接口的稀疏结构强化学习任务，我们发现大模型生成的奖励函数设计应被视为调试而非一次性生成。以MiniGrid为核心评估环境，MuJoCo作为边界压力测试，审计揭示了两种主要的一次性失败模式——奖励泛滥与语义/API误解，以及较少见的弱塑造情况。提出诊断驱动的迭代优化方法，利用训练诊断与失败模式分类指导针对性修正。该方法使DoorKey-8x8成功率从2.3%提升至97.6%，KeyCorridor从31.2%升至86.7%，且在多种子实验中表现一致。对照实验表明，收益并非来自重试或额外训练：仅改写提示会导致性能大幅下降，而静态词汇控制恢复了大部分差距（87.6%、70.7%），说明分类提示是核心机制，动态标签仅提供部分增量证据。预算匹配与Best-of-3对比分离出优化与选择/训练时间效应。组件移除测试、敏感性分析及作者标签审计提供了收敛支持，确认调试解释的有效性并揭示校准局限。连续控制结果表明，基于成功的诊断在密集奖励运动任务中可能误报，返回趋势反馈虽消除一个假阳性机制，但未带来稳健增益。低调用协议是与群体搜索的成本对比，非基准比较。在四个交叉方差设计环境中，点估计显示当大模型奖励函数方差占主导时收益更大，但置信区间较宽。该方法限于具备可靠接口的稀疏结构任务，在PPO框架下有效；事件文本等场景可能有益、有害或中性。
原文摘要 · Abstract (English)
For sparse, structured reinforcement-learning tasks with semantic reward-function interfaces, LLM-generated reward shaping is better framed as debugging than one-shot generation. We study PPO-trained agents using MiniGrid as core evaluation and MuJoCo as boundary stress test. Our audit finds two dominant one-shot failure modes -- reward flooding and semantic/API misunderstanding -- plus a rarer weak-shaping case. We propose diagnostic-driven iterative refinement, where training diagnostics and a failure-mode taxonomy guide targeted reward-function revision. Refinement improves DoorKey-8x8 from 2.3% to 97.6% and KeyCorridor from 31.2% to 86.7% with high seed-to-seed variance. Controls show these gains are not from retrying or extra training: metrics-only re-prompting yields large drops, while a static-vocabulary control recovers much of the gap (87.6%; 70.7%), showing the taxonomy prompt is a major mechanism and dynamic labels provide only partially isolated incremental evidence. Budget-matched and Best-of-3 comparisons separate refinement from selection and training-time effects. Component-removal tests, sensitivity analyses, and an audit against author labels provide converging evidence for the debugging interpretation while revealing calibration limits. Continuous-control results show the boundary: success-based diagnostics can misfire in dense-reward locomotion, and return-trend feedback removes one false-positive mechanism without robust gains. The low-call protocol is a cost contrast with population-based reward search, not a benchmark comparison. In four crossed-variance-design environments, point estimates suggest larger gains when LLM reward-function variance dominates but bootstrap intervals are wide. The method is bounded to sparse structured tasks with reliable interfaces under PPO; fields like event_text may help, hurt, or be neutral.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。