用分位数损失改进强化学习中的过估计问题
Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss
- 提出基于下分位数的温和目标，抑制Q值过估计
- 在DDPG、SAC等算法中显著降低过估计偏差
- 可无缝集成到主流强化学习框架，适合算法优化者
模型无关强化学习（MF-RL）中的过估计是时序差分学习和Q函数近似固有的问题。为解决该问题，我们提出一种新的Q函数更新目标，通过凸优化高估的Q值及其下界来实现。核心贡献在于利用状态条件下的Q值分布的下分位数高效估计该下界。所提温和目标可无缝融入当前最先进的MF-RL算法，包括深度确定性策略梯度（DDPG）和软动作价值批评家（SAC）。实验验证了该方法在DDPG、SAC及分布式强化学习算法中有效缓解过估计偏差。
原文摘要 · Abstract (English)
Overestimation is a fundamental characteristic of model-free reinforcement learning (MF-RL), arising from the principles of temporal difference learning and the approximation of the Q-function. To address this challenge, we propose a novel moderate target in the Q-function update, formulated as a convex optimization of an overestimated Q-function and its lower bound. Our primary contribution lies in the efficient estimation of this lower bound through the lower expectile of the Q-value distribution conditioned on a state. Notably, our moderate target integrates seamlessly into state-of-the-art (SOTA) MF-RL algorithms, including Deep Deterministic Policy Gradient (DDPG) and Soft Actor Critic (SAC). Experimental results validate the effectiveness of our moderate target in mitigating overestimation bias in DDPG, SAC, and distributional RL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。