证明了平均奖励RL中局部学习率可收敛，全局学习率却可能发散
On the Divergence of Differential Temporal Difference Learning without Local Clocks
- 用局部/全局时钟区分两种学习率，分析其收敛性差异
- 构造反例：局部时钟收敛时，全局时钟可能导致发散
- 解决平均奖励RL中的长期开放问题，适合强化学习研究者
学习率是强化学习中的关键组件。本文通过全局时钟（仅依赖时间步t）和局部时钟（依赖状态访问次数ν(s,t)）区分两类学习率。在折扣型强化学习中，两者收敛性等价，且未发现反例。但本工作首次证明，在平均奖励强化学习中，这种对应关系不成立：构造的反例显示，差分时序差分学习在局部时钟下收敛，但在全局时钟下可能发散。该结果解决了Wan等人[2021]、Blaser等人[2026]提出的开放问题。
原文摘要 · Abstract (English)
Learning rate is a critical component of reinforcement learning (RL). This work uses global and local clocks to distinguish two types of learning rates. The former is of the standard form $α_t$ that depends only on the time step $t$ (i.e., a global clock). The latter is of the form $α_{ν(S_t, t)}$, where $ν(s, t)$ counts the number of visits to state $s$ until time $t$ (i.e., a local clock). In discounted RL, an RL algorithm that is convergent with a local clock is always also convergent with a global clock, and vice versa. We are not aware of any counterexample. The key contribution of this work is to show that this nice correspondence breaks down in average-reward RL. Specifically, we construct a counterexample showing that although differential temporal difference learning is convergent with a local clock, it can diverge with a global clock. This counterexample closes the open problem in Wan et al. [2021], Blaser et al. [2026].
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。