首次分析奖励动态变化下的演员-评论家算法，证明其可实现与静态奖励相同的收敛速度。
Finite-time Convergence Analysis of Actor-Critic with Evolving Reward
- 在马尔可夫采样下，研究每步可变的奖励函数对算法的影响。
- 证明当奖励变化足够慢时，算法误差以 $O(1/ ext{sqrt}{T})$ 收敛。
- 适用于奖励塑形、课程学习等常用技巧，为实际RL方法提供理论支持。
许多流行的强化学习（RL）算法使用动态奖励函数，如奖励塑形、熵正则化或课程学习，但其理论基础仍不完善。本文首次在马尔可夫采样下，对单时间尺度的演员-评论家算法在动态奖励条件下的有限时间收敛性进行了分析。考虑奖励参数在每一步可能变化的情形，该变化同时影响策略优化与价值估计。在标准假设下，我们推导出演员与评论家误差的非渐近界。结果表明，若奖励参数变化足够缓慢，则可达到 $O(1/ ext{sqrt}{T})$ 的收敛速率，与静态奖励下的最优速率一致。当奖励通过梯度规则更新且梯度有界、与演员和评论家同频更新时，该速率依然保持，为多种主流强化学习技术提供了理论依据。作为次要贡献，我们提出了马尔可夫采样下分布偏移的新分析，使静态奖励情形下的最优速率提升了一个 $ ext{log}^2 T$ 因子。
原文摘要 · Abstract (English)
Many popular practical reinforcement learning (RL) algorithms employ evolving reward functions-through techniques such as reward shaping, entropy regularization, or curriculum learning-yet their theoretical foundations remain underdeveloped. This paper provides the first finite-time convergence analysis of a single-timescale actor-critic algorithm in the presence of an evolving reward function under Markovian sampling. We consider a setting where the reward parameters may change at each time step, affecting both policy optimization and value estimation. Under standard assumptions, we derive non-asymptotic bounds for both actor and critic errors. Our result shows that an $O(1/\sqrt{T})$ convergence rate is achievable, matching the best-known rate for static rewards, provided the reward parameters evolve slowly enough. This rate is preserved when the reward is updated via a gradient-based rule with bounded gradient and on the same timescale as the actor and critic, offering a theoretical foundation for many popular RL techniques. As a secondary contribution, we introduce a novel analysis of distribution mismatch under Markovian sampling, improving the best-known rate by a factor of $\log^2T$ in the static-reward case.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。