通过优化判别网络的条件数,提升强化学习样本效率。
XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
- 结合批量归一化、权重归一化与分布交叉熵损失,改善优化景观。
- 在55个本体感觉和15个视觉连续控制任务上达最优样本效率。
- 参数量更少,适合资源受限场景下的高效强化学习应用。
样本效率是有效深度强化学习算法的核心特性。近期工作通过增加复杂性(如更大模型、特殊网络结构、更复杂算法)来提升性能,但多基于经验驱动。本文从更严谨的角度出发,聚焦判别网络的优化景观,利用判别器海森矩阵的特征谱和条件数,系统分析常见架构设计对训练动态的影响。结果表明,批量归一化(BN)、权重归一化(WN)与分布交叉熵(CE)损失的组合可使条件数比基线小多个数量级。该组合还能自然约束梯度范数,对非平稳目标与自举过程中的稳定有效学习率至关重要。基于此,我们提出XQC:一种基于软演员-评论家、具备优化感知原则的高效深度演员-评论家算法。在55个本体感觉与15个视觉连续控制任务中均达到当前最优样本效率,且参数量显著低于竞争方法。代码见 danielpalenicek.github.io/projects/xqc。
原文摘要 · Abstract (English)
Sample efficiency is a central property of effective deep reinforcement learning algorithms. Recent work has improved this through added complexity, such as larger models, exotic network architectures, and more complex algorithms, which are typically motivated purely by empirical performance. We take a more principled approach by focusing on the optimization landscape of the critic network. Using the eigenspectrum and condition number of the critic's Hessian, we systematically investigate the impact of common architectural design decisions on training dynamics. Our analysis reveals that a novel combination of batch normalization (BN), weight normalization (WN), and a distributional cross-entropy (CE) loss produces condition numbers orders of magnitude smaller than baselines. This combination also naturally bounds gradient norms, a property critical for maintaining a stable effective learning rate under non-stationary targets and bootstrapping. Based on these insights, we introduce XQC: a well-motivated, sample-efficient deep actor-critic algorithm built upon soft actor-critic that embodies these optimization-aware principles. We achieve state-of-the art sample efficiency across 55 proprioception and 15 vision-based continuous control tasks, all while using significantly fewer parameters than competing methods. Our code is available at danielpalenicek.github.io/projects/xqc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。