提出几何动作控制新方法,让强化学习更高效地处理连续动作空间。
Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learning
- 将动作生成分解为方向与可学习浓度参数,简化计算流程。
- 在六个基准上表现超越或媲美顶尖方法,复杂任务提升超112%。
- 适合关注连续控制效率与几何结构的算法研究者。
高斯策略在深度强化学习中主导连续控制,但其无界支持与有界动作空间存在根本矛盾,需借助人工挤压函数,破坏动作空间几何结构。尽管冯·米塞斯-费舍尔(vMF)分布理论上更合适,但依赖贝塞尔函数和拒绝采样,难以实用。本文提出几何动作控制(GAC),保留球面分布的几何优势,同时简化计算:将动作生成分解为方向向量与可学习浓度参数,实现确定性动作与均匀球面噪声间的高效插值。参数量从2d降至d+1,避免了vMF的O(dk)复杂度,仅需O(d)操作。实验表明,GAC在六个MuJoCo基准上稳定达到或超过当前最优性能,相较SAC在Ant-v4上提升37.6%,在复杂DMControl任务中最高达112%。消融分析显示,球面归一化与自适应浓度控制对性能至关重要。结果表明,高效连续控制的关键在于尊重动作空间几何,而非复杂分布。
原文摘要 · Abstract (English)
Gaussian policies have dominated continuous control in deep reinforcement learning (RL), yet they suffer from a fundamental mismatch: their unbounded support requires ad-hoc squashing functions that distort the geometry of bounded action spaces. While von Mises-Fisher (vMF) distributions offer a theoretically grounded alternative on the sphere, their reliance on Bessel functions and rejection sampling hinders practical adoption. We propose \textbf{Geometric Action Control (GAC)}, a novel action generation paradigm that preserves the geometric benefits of spherical distributions while \textit{simplifying computation}. GAC decomposes action generation into a direction vector and a learnable concentration parameter, enabling efficient interpolation between deterministic actions and uniform spherical noise. This design reduces parameter count from \(2d\) to \(d+1\), and avoids the \(O(dk)\) complexity of vMF rejection sampling, achieving simple \(O(d)\) operations. Empirically, GAC consistently matches or exceeds state-of-the-art methods across six MuJoCo benchmarks, achieving 37.6\% improvement over SAC on Ant-v4 and up to 112\% on complex DMControl tasks, demonstrating strong performance across diverse benchmarks. Our ablation studies reveal that both \textbf{spherical normalization} and \textbf{adaptive concentration control} are essential to GAC's success. These findings suggest that robust and efficient continuous control does not require complex distributions, but a principled respect for the geometry of action spaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。