通过跨动作共享价值加速Q-learning，减少学习延迟与过估计。
Accelerating Q-learning through Efficient Value-Sharing across Actions
- 引入均值扩展层，跨动作共享状态内价值信息
- 在57个Atari游戏上提升整体性能，减少价值过估计
- 无需修改算法，可直接嵌入现有Q网络结构
动作值是Q-learning等控制算法的基础，但学习过程常因初始值接近零而缓慢收敛至真实值。传统方法独立更新每个状态-动作对，未利用同一状态下各动作间的共性。本文提出均值扩展层，通过共享状态内动作间的价值信息，将学习目标从直接预测高幅值动作值转化为低范数表示，从而加速收敛。该模块在深度强化学习中作为无参数附加层，可无缝集成于DQN与隐式分位数网络架构，不改变原有算法。实验表明，在57个Atari 2600游戏中显著提升综合性能，增强动作差距，大幅降低价值过估计现象。
原文摘要 · Abstract (English)
Action values are foundational to many control algorithms such as Q-learning. Therefore, efficient action-value learning is central to reinforcement learning (RL). However, learning them can be slow, requiring many updates to move values from their initialization, typically near zero, to their true values, which may be far from zero. Moreover, action-value learning algorithms typically update each state-action pair independently, without learning a value that is common to all actions within a state. In this paper, we address these inefficiencies by introducing the mean-expansion layer, which accelerates action-value learning by sharing values across actions within a state and by changing the problem from directly learning potentially large action-values to learning a lower-norm representation of them. In deep RL, this layer can be applied as a parameter-free addition to Q-network architectures without altering the underlying algorithm. Applied to deep Q-networks and implicit quantile networks, it improves aggregate performance across 57 Atari 2600 games while increasing action gaps and dramatically reducing value overestimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。