改进离策略预测的收敛速度,用行为策略信息优化更新几何。
Behavior-Induced Mirror-Prox Temporal-Difference Learning for Faster Off-Policy Prediction

- 用行为策略贝尔曼矩阵对称部分替代协方差度量,构建新梯度方法。
- 在两状态、随机游走等基准上验证,收敛速度优于GTD2-MP。
- 适合需要快速稳定离策略预测的研究者,如强化学习算法设计。
梯度时序差分方法在使用线性函数逼近时能提供稳定的离策略预测,但其实际性能受辅助变量度量所诱导的几何结构影响显著。现有镜像-近端时序差分方法通常采用特征协方差度量,而混合型TD方法表明行为策略转移信息可提供更优的更新几何。本文提出一种行为诱导的镜像-近端时序差分方法(STHTD-MP),将原对偶鞍点公式中的协方差度量替换为行为策略贝尔曼矩阵的对称部分。该方法仅需一个学习率,对所得混合鞍点算子应用镜像-近端预测-校正步骤。在标准随机逼近假设下,我们提供了固定策略线性预测的收敛性分析:行为诱导度量正定,联合均值系统为赫尔维茨型,有界性由李雅普诺夫论证保证,随机递归通过常微分方程法收敛。进一步推导出投影-预言机的遍历间隙界,并基于确定性镜像-近端误差矩阵的谱半径,与GTD2-MP进行精确均值算子比较。分析表明,当行为诱导度量改善鞍点几何时,STHTD-MP的均值收缩因子可小于GTD2-MP。两状态、随机游走和Boyan Chain基准的精确数值均值算子分析支持此结论,而Baird反例被识别为严格假设失效的奇异边界情形。
原文摘要 · Abstract (English)
Gradient temporal-difference methods provide stable off-policy prediction with linear function approximation, but their practical performance is strongly affected by the geometry induced by the auxiliary-variable metric. Existing Mirror-Prox TD methods typically use the feature covariance metric, whereas hybrid TD methods suggest that behavior-policy transition information can provide a more informative update geometry. This paper proposes a behavior-induced Mirror-Prox temporal-difference method, called STHTD-MP, which replaces the covariance metric in the primal-dual saddle-point formulation with the symmetric part of the behavior-policy Bellman matrix. The method keeps a single learning rate for the primal and auxiliary variables and applies a Mirror-Prox prediction-correction step to the resulting hybrid saddle-point operator. We provide a formal convergence analysis for fixed-policy linear prediction under standard stochastic approximation assumptions: the behavior-induced metric is positive definite, the joint mean system is Hurwitz, boundedness follows from a Lyapunov argument, and the stochastic recursion converges by the ODE method. We further derive projected-oracle ergodic gap bounds and an exact mean-operator comparison with GTD2-MP based on the spectral radius of the deterministic Mirror-Prox error matrix. The analysis shows that STHTD-MP can have a smaller mean contraction factor than GTD2-MP when the behavior-induced metric improves the saddle-point geometry. Exact numerical mean-operator analysis on two-state, Random Walk, and Boyan Chain benchmarks supports this condition, while Baird's counterexample is identified as a singular boundary case where the strict assumptions fail.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。