让强化学习的折扣因子随状态变化，提升决策效果
AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning
- 设计可学习的状态相关折扣函数，结合返回一致性约束防止崩溃
- 在连续控制任务中优于固定折扣率，线上测试显著提升性能
- 适合需要动态规划长度的复杂强化学习场景
强化学习中的折扣因子决定了有效规划范围和自举强度，但大多数深度强化学习方法在所有状态下使用单一固定值。虽然状态依赖折扣在理论上具有吸引力，但简单的深度演员-评论家实现可能导致不稳定并退化为TD误差塌陷。我们提出AdaGamma，一种实用的深度演员-评论家方法，用于状态依赖折扣，通过联合学习状态依赖折扣函数和返回一致性目标来正则化诱导的备份结构。理论上，我们分析了状态依赖折扣所诱导的贝尔曼算子，并在适当条件下建立了其基本适定性。实证上，AdaGamma可集成于SAC与PPO，在连续控制基准测试中持续提升性能，并在京东物流平台的在线A/B测试中取得统计显著增益。结果表明，当与防止目标操纵的返回一致性目标结合时，状态依赖折扣可在深度强化学习中有效实现。
原文摘要 · Abstract (English)
The discount factor in reinforcement learning controls both the effective planning horizon and the strength of bootstrapping, yet most deep RL methods use a single fixed value across all states. While state-dependent discounting is conceptually appealing, naive deep actor--critic implementations can become unstable and degenerate toward TD-error collapse. We propose AdaGamma, a practical deep actor--critic method for state-dependent discounting that learns a state-dependent discount function together with a return-consistency objective to regularize the induced backup structure. On the theory side, we analyze the Bellman operator induced by state-dependent discounting and establish its basic well-posedness properties under suitable conditions. Empirically, AdaGamma integrates into both SAC and PPO, yielding consistent improvements on continuous-control benchmarks, and achieves statistically significant gains in an online A/B test on the JD Logistics platform. These results suggest that state-dependent discounting can be made effective in deep RL when coupled with a return-consistency objective that prevents degenerate target manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。