揭示Q-learning中目标更新频率的最优策略，可显著提升学习效率。
The Role of Target Update Frequencies in Q-Learning
- 将目标更新建模为嵌套优化，用近似动态规划分析其机制
- 证明固定更新周期会引入对数级样本复杂度开销
- 提出随训练进程几何增长的自适应更新策略更优
目标网络更新频率（TUF）是（深度）Q-learning中的核心稳定机制。然而其选择仍缺乏理论理解，常被当作普通超参数而非有原则的设计决策。本文通过近似动态规划视角，对表格型Q-learning中的目标固定进行理论分析。将周期性目标更新建模为嵌套优化框架，外层每次迭代应用一个由通用内循环优化器近似的非精确贝尔曼最优算子。严格理论推导出异步采样设定下的有限时间收敛性，内层特化为随机梯度下降。结果揭示了目标更新周期引发的偏差-方差权衡，明确给出了该关键超参数的最优设置方法。证明恒定目标更新策略次优，会引入对数级样本复杂度开销，而自适应策略可完全避免。分析表明最优目标更新频率应随学习过程几何增长。
原文摘要 · Abstract (English)
The target network update frequency (TUF) is a central stabilization mechanism in (deep) Q-learning. However, their selection remains poorly understood and is often treated merely as another tunable hyperparameter rather than as a principled design decision. This work provides a theoretical analysis of target fixing in tabular Q-learning through the lens of approximate dynamic programming. We formulate periodic target updates as a nested optimization scheme in which each outer iteration applies an inexact Bellman optimality operator, approximated by a generic inner loop optimizer. Rigorous theory yields a finite-time convergence analysis for the asynchronous sampling setting, specializing to stochastic gradient descent in the inner loop. Our results deliver an explicit characterization of the bias-variance trade-off induced by the target update period, showing how to optimally set this critical hyperparameter. We prove that constant target update schedules are suboptimal, incurring a logarithmic overhead in sample complexity that is entirely avoidable with adaptive schedules. Our analysis shows that the optimal target update frequency increases geometrically over the course of the learning process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。