为带函数逼近的熵正则Q学习提供了高维中心极限定理的收敛速率分析。
On Gaussian approximation for entropy-regularized Q-learning with function approximation
- 用线性近似和多项式步长分析熵正则Q学习的迭代收敛性。
- 在样本数n下，高斯近似误差为n^{-1/4}量级（含对数因子）。
- 适用于研究强化学习算法统计性质的研究者，尤其关注收敛性分析。
本文推导了熵正则异步Q学习结合线性函数逼近及多项式步长k^{-ω}（ω∈(1/2,1)）所生成的Polyak-Ruppert平均迭代在高维中心极限定理中的收敛速率。假设观测三元组序列(s_k,a_k,s_{k+1})_{k≥0}构成均匀几何遍历马尔可夫链，并在投影软贝尔曼方程满足适当正则性条件下，建立了凸距离下的高斯近似界，其速率为n^{-1/4}量级（含关于n的对数多项式因子），其中n为算法使用的样本数。为获得该结果，将软贝尔曼递归线性化，并对主导鞅项进行高斯近似。最后，还推导出算法末次迭代的高阶矩界，可能具有独立研究价值。
原文摘要 · Abstract (English)
In this paper, we derive rates of convergence in the high-dimensional central limit theorem for Polyak--Ruppert averaged iterates generated by entropy-regularized asynchronous Q-learning with linear function approximation and a polynomial stepsize $k^{-ω}$, $ω\in (1/2,1)$. Assuming that the sequence of observed triples $(s_k,a_k,s_{k+1})_{k \geq 0}$ forms a uniformly geometrically ergodic Markov chain, and under suitable regularity conditions for the projected soft Bellman equation, we establish a Gaussian approximation bound in the convex distance with rate of order $n^{-1/4}$, up to polylogarithmic factors in $n$, where $n$ is the number of samples used by the algorithm. To obtain this result, we combine a linearization of the soft Bellman recursion with a Gaussian approximation for the leading martingale term. Finally, we derive high-order moment bounds for the algorithm's last iterate, which might be of independent interest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。