不使用策略梯度也能在复杂动作空间中高效学习
Learning in complex action spaces without policy gradients
- 提出QMLE框架，用最大似然估计改进Q-learning
- 在复杂动作空间中计算成本与策略梯度相当
- 在DeepMind控制套件上表现媲美顶尖方法
尽管传统观点认为策略梯度方法更适合复杂动作空间，但基础研究已证明两种范式在小规模离散动作空间中等价（O'Donoghue et al., 2017; Schulman et al., 2017a）。这引发疑问：为何随着动作空间复杂度上升，两者的适用性与性能出现分化？我们假设这种看似优势源于普遍原则，而非方法本身，且这些原则同样适用于动作值方法。本文识别出三项通用原则，并构建框架将其融入动作值方法。为验证假设，我们提出QMLE（Q-learning with maximum likelihood estimation）作为实例。结果表明，QMLE可在复杂动作空间中以接近策略梯度的计算成本运行，且无需使用策略梯度。此外，其在DeepMind Control Suite上表现优异，甚至超越如DMPO和D4PG等先进方法。
原文摘要 · Abstract (English)
While conventional wisdom holds that policy gradient methods are better suited to complex action spaces than action-value methods, foundational work has shown that the two paradigms are equivalent in small, finite action spaces (O'Donoghue et al., 2017; Schulman et al., 2017a). This raises the question of why their computational applicability and performance diverge as the complexity of the action space increases. We hypothesize that the apparent superiority of policy gradients in such settings stems not from intrinsic qualities of the paradigm but from universal principles that can also be applied to action-value methods, enabling similar functions. We identify three such principles and provide a framework for incorporating them into action-value methods. To support our hypothesis, we instantiate this framework in what we term QMLE, for Q-learning with maximum likelihood estimation. Our results show that QMLE can be applied to complex action spaces at a computational cost comparable to that of policy gradient methods, all without using policy gradients. Furthermore, QMLE exhibits strong performance on the DeepMind Control Suite, even when compared to state-of-the-art methods such as DMPO and D4PG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。