对比双值函数TD方法，发现一种新算法在控制任务中优于经典Q-learning。
An Analysis of Action-Value Temporal-Difference Methods That Learn State Values
- 用两个不对称值函数（状态值+动作值）分步学习，替代单一动作值函数。
- 在控制任务中，仅AV-learning方法比Q-learning有显著提升，且样本效率更高。
- 提出新算法RDQ，MinAtar上大幅超越原始Dueling DQN。
时序差分（TD）学习的核心是自举：利用价值预测生成新的价值预测。大多数控制类TD方法通过单个动作值函数（如Q-learning和Sarsa）进行策略学习。相比之下，较少研究关注从两个非对称值函数自举的方法——即先学习状态值，再用于学习动作值。现有方法可分为QV-learning与AV-learning两类。尽管已有初步研究，但尚不明确这类方法何时更优，以及其理论是否普遍成立。本文分析了这两类算法的收敛性与样本效率。结果表明，在预测任务中两者均优于期望Sarsa，但在控制任务中仅AV-learning方法能显著超越Q-learning。最后，我们提出一种新型AV-learning算法——正则化对偶Q学习（RDQ），在MinAtar基准测试中显著优于对偶DQN。
原文摘要 · Abstract (English)
The hallmark feature of temporal-difference (TD) learning is bootstrapping: using value predictions to generate new value predictions. The vast majority of TD methods for control learn a policy by bootstrapping from a single action-value function (e.g., Q-learning and Sarsa). Significantly less attention has been given to methods that bootstrap from two asymmetric value functions: i.e., methods that learn state values as an intermediate step in learning action values. Existing algorithms in this vein can be categorized as either QV-learning or AV-learning. Though these algorithms have been investigated to some degree in prior work, it remains unclear if and when it is advantageous to learn two value functions instead of just one -- and whether such approaches are theoretically sound in general. In this paper, we analyze these algorithmic families in terms of convergence and sample efficiency. We find that while both families are more efficient than Expected Sarsa in the prediction setting, only AV-learning methods offer any major benefit over Q-learning in the control setting. Finally, we introduce a new AV-learning algorithm called Regularized Dueling Q-learning (RDQ), which significantly outperforms Dueling DQN in the MinAtar benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。