用方差最小化加速强化学习收敛，提升训练效率。
A Variance Minimization Approach to Temporal-Difference Learning
- 以方差最小化替代传统误差最小化，优化学习路径
- 新算法在多个测试环境上收敛速度显著快于基线方法
- 适合追求快速收敛的强化学习应用开发
快速收敛是当前强化学习的重要需求。在线性函数逼近框架下，关键矩阵最小特征值的大小直接影响收敛速度。传统基于价值的强化学习算法聚焦于误差最小化，本文提出一种方差最小化（VM）方法，替代误差最小化。基于此，提出两个目标：贝尔曼误差方差（VBE）与投影贝尔曼误差方差（VPBE），并推导出VMTD、VMTDC和VMETD算法。论文提供了算法收敛性及最优策略不变性的理论证明。实验验证了所提算法的有效性，在多个基准任务上展现出更快的收敛速度。
原文摘要 · Abstract (English)
Fast-converging algorithms are a contemporary requirement in reinforcement learning. In the context of linear function approximation, the magnitude of the smallest eigenvalue of the key matrix is a major factor reflecting the convergence speed. Traditional value-based RL algorithms focus on minimizing errors. This paper introduces a variance minimization (VM) approach for value-based RL instead of error minimization. Based on this approach, we proposed two objectives, the Variance of Bellman Error (VBE) and the Variance of Projected Bellman Error (VPBE), and derived the VMTD, VMTDC, and VMETD algorithms. We provided proofs of their convergence and optimal policy invariance of the variance minimization. Experimental studies validate the effectiveness of the proposed algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。