提出隐式更新方法，让平均奖励TD学习更稳定且无需精细调参。
Implicit Updates for Average-Reward Temporal Difference Learning
- 用隐式固定点更新替代显式更新，实现自适应稳定性。
- 在更宽松的学习率条件下仍保持有限时间误差界。
- 适合需要稳定训练的强化学习任务，尤其对调参敏感场景。
时序差分（TD）学习是强化学习的核心。在平均奖励设置下，标准TD(λ)对学习率极为敏感，需精细调参以保证数值稳定。本文提出平均奖励隐式TD(λ)，通过隐式固定点更新实现数据自适应稳定，同时保持与标准方法相同的每轮计算复杂度。相比以往针对平均奖励TD(λ)的有限时间分析所施加的严格学习率限制，本方法在更弱的学习率条件下建立了有限时间误差界。实验表明，隐式版本在更广的学习率范围内运行可靠，数值稳定性显著提升，从而更高效地支持策略评估与学习，展现出对标准方法的有效替代性。
原文摘要 · Abstract (English)
Temporal difference (TD) learning is a cornerstone of reinforcement learning. In the average-reward setting, standard TD($λ$) is highly sensitive to the choice of step-size and thus requires careful tuning to maintain numerical stability. We introduce average-reward implicit TD($λ$), which employs an implicit fixed point update to provide data-adaptive stabilization while preserving the per iteration computational complexity of standard average-reward TD($λ$). In contrast to prior finite-time analyses of average-reward TD($λ$), which impose restrictive step-size conditions, we establish finite-time error bounds for the implicit variant under substantially weaker step-size requirements. Empirically, average-reward implicit TD($λ$) operates reliably over a much broader range of step-sizes and exhibits markedly improved numerical stability. This enables more efficient policy evaluation and policy learning, highlighting its effectiveness as a robust alternative to average-reward TD($λ$).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。