提出一种无需小折扣因子的鲁棒强化学习算法，实现线性函数逼近下的有限时间收敛。
Finite-Time Convergence of Distributionally Robust Q-Learning with Linear Function Approximation
- 结合目标网络与双函数近似，处理χ²分布不确定性下的鲁棒贝尔曼更新。
- 在单个马尔可夫轨迹下，实现对最优鲁棒Q值的有限时间收敛，误差可控。
- 适用于实际部署中环境变化场景，适合关注鲁棒性的强化学习研究者。
分布鲁棒强化学习（DRRL）旨在寻找在部署时转移模型与数据生成的名义模型不同时仍表现良好的策略。现有大多数有限样本保证局限于表格型、基于模型或需要生成访问，或仅在特定结构（如线性转移模型）下获得函数逼近结果，且常需小折扣因子假设。本文研究在$(s,a)$-矩形χ²不确定性集下，使用线性函数逼近鲁棒Q函数的免模型鲁棒Q-learning，仅需来自未知名义模型的单一马尔可夫轨迹。算法采用目标网络外循环与双函数逼近方案实现χ²鲁棒贝尔曼更新。双过程使用动量跟踪评论家、后缀平均、新评估阶段处理方差类动量，并引入可调平滑参数以获得Lipschitz连续的χ²对偶梯度。我们证明了在无小折扣因子假设下，算法能在有限时间内收敛至最优鲁棒Q函数，误差不超过近似误差。结果填补了鲁棒强化学习算法经验应用与非渐近保证之间的差距。
原文摘要 · Abstract (English)
Distributionally robust reinforcement learning (DRRL) seeks policies that perform well when the deployment transition model differs from the nominal model generating the data. Most finite-sample guarantees for DRRL are tabular, model-based, rely on generative access, or obtain function-approximation guarantees only under additional structure, such as linear-transition models or restrictive discount-factor conditions. We study discounted model-free robust Q-learning under an $(s,a)$-rectangular chi-square uncertainty set, with linear approximation of the robust Q-function, using only a single Markovian trajectory from an unknown nominal model. Our algorithm combines a target-network outer loop with a dual function-approximation scheme for the chi-square robust Bellman update. The dual procedure uses moment-tracking critics, suffix averaging, a fresh-evaluation stage for the variance-like moment, and a tunable smoothing parameter to have a Lipschitz-continuous chi-square dual gradient. We prove a finite-time convergence bound to the optimal robust Q-function up to approximation error, without imposing a small-discount-factor assumption. Our results help close a gap between the empirical use of robust RL algorithms and the non-asymptotic guarantees available for their non-robust counterparts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。