研究连续时间多臂老虎机在扩散环境下的策略梯度收敛与后悔，证明其几乎必然收敛到最优臂。
Convergence and Regret of the Policy Gradient for Multi-Armed Bandits in Diffusion Environment
- 使用logit参数化策略，结合随机微分方程框架分析梯度更新
- 常数学习率下后悔上界为O(log T)，且低于阈值时可保证收敛
- 提出新Lyapunov函数，提升分析透明性，适用于离散与连续场景
本文在连续时间强化学习框架下，研究由随机微分方程（SDE）描述的扩散环境中多臂老虎机问题的策略梯度更新。采用对数几率（logit）参数化随机策略，证明在任意常数学习率下，算法几乎必然收敛至最优臂。进一步推导出当常数学习率低于一个与时间无关的阈值时，非渐近后悔上界为O(log T)。相较于Lattimore（2026a）的工作，通过构建新的Lyapunov函数改进了分析方法，并展示了利用SDE工具分析策略梯度的清晰性。该方法同样适用于离散时间策略梯度算法的分析。
原文摘要 · Abstract (English)
This paper studies the policy gradient update for a multi-arm bandit problem in diffusion environment that is described by a stochastic differential equation (SDE) under the continuous-time reinforcement learning framework by Wang et al. (2020), Jia and Zhou (2022b). With the logit parameterization for the stochastic policy, we show that it converges almost surely to the optimal arm under an arbitrary constant learning rate. Furthermore, we derive the non-asymptotic regret upper bound when the constant learning rate is below a time-invariant threshold; and the regret bound has order $O(\log T)$. We improve the analysis in Lattimore (2026a) for the same SDE by constructing a novel Lyapunov function and demonstrate the transparency of analyzing policy gradient using the tools in SDEs. In addition, the same Lyapunov function is also helpful in analyzing the discrete-time policy gradient algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。