解决强化学习中两类参数更新的收敛性问题，适用于更真实的马尔可夫噪声场景。
Convergence of Two-Timescale Markovian Stochastic Approximations with Applications in Reinforcement Learning
- 用运行最大值控制快速参数，提升算法稳定性。
- 首次证明离策略线性近似下TDC方法几乎必然收敛。
- 无需投影算子或紧凑噪声空间，适用范围更广。
本文研究了双时标随机逼近（two-timescale stochastic approximations, SA）的收敛性，这类算法在强化学习中用于分别以快慢时标更新两组参数。典型例子包括带梯度修正的时间差分学习（TDC）和演员-评论家方法。以往结果仅在独立同分布（i.i.d.）噪声下成立，本文首次在更贴近强化学习实际的马尔可夫噪声设定下建立了稳定性与收敛性。关键创新在于用慢速参数的运行最大值而非当前值来控制快速参数，避免使用投影算子且不要求噪声位于紧致空间。作为核心应用，本文首次证明了在离策略学习与线性函数近似下，带有资格迹的TDC方法几乎必然收敛。
原文摘要 · Abstract (English)
This work studies the convergence of two-timescale stochastic approximations (SA), a class of iterative algorithms that update two sets of parameters in fast and slow timescales respectively. Notable examples of two-timescale SA in reinforcement learning (RL) include temporal difference learning with gradient correction (TDC) and actor-critic methods. Previously, the stability (i.e., boundedness) and convergence of two-timescale SA were only established under i.i.d. noise. This work instead establishes the stability and convergence of two-timescale SA under Markovian noise, a setup that is more realistic in RL. Notably, we do not need to use any projection operator and the noise does not need to live in a compact space. Our key technical novelty is to control the fast timescale parameter with the running max of the slow timescale parameter, instead of with the current slow timescale parameter, as most prior works do. As a key application, we establish the first almost sure convergence of TDC with eligibility traces under off-policy learning with linear function approximation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。