利用反射对称性提升状态空间强化学习的样本效率
Reflex: Reinforcement Learning with Reflection Symmetry Exploitation in State-Based Continuous Control
- 通过反射对称性正则化，改进策略学习机制
- 在多个基准任务上实现更优性能与更高采样效率
- 适用于连续控制场景，兼容主流强化学习算法
强化学习长期面临样本效率低的问题。一种有前景的解决思路是利用群不变马尔可夫决策过程（G-不变MDP）。现有工作主要聚焦于基于图像的强化学习和旋转对称性（如SO(2)），而对基于状态的强化学习和反射对称性的研究仍不充分。本文聚焦于基于状态的连续控制任务，提出Reflex框架，无缝集成于在线与离线强化学习算法中。我们形式化了两类反射：轴向反射与双边反射，并刻画其对应的变换关系。基于对称性保持最优价值函数与策略的理论分析，Reflex通过原则性的对称性正则化机制将反射对称性融入策略学习。我们将Reflex与PPO和SAC结合，在OpenAI Gym和DeepMind Control一系列基准上进行评估，结果表明其性能优于标准基线，同时显著提升样本效率。代码已开源：https://github.com/TonyStark042/Reflex。
原文摘要 · Abstract (English)
Reinforcement learning has long struggled with poor sample efficiency. One promising approach to mitigate this problem is leveraging group-invariant Markov Decision Processes ($G$-invariant MDPs). Existing works in this direction have primarily focused on image-based RL and rotational symmetry such as $\mathrm{SO(2)}$, leaving state-based RL and reflection symmetry largely underexplored. In this work, we focus on state-based continuous control tasks and exploit reflection symmetry by introducing Reflex, a paradigm that seamlessly integrates with both on-policy and off-policy RL algorithms. We formalize two types of reflection-axial reflection and bilateral reflection, and characterize their corresponding transformations. Building on a theoretical analysis of symmetry-preserving optimal value functions and policies, Reflex integrates reflection symmetry into policy learning through principled symmetry regularization mechanisms. We integrate Reflex with PPO and SAC, and evaluate it on a suite of OpenAI Gym and DeepMind Control benchmarks, demonstrating superior performance over standard baselines while improving sample efficiency. Our code is available at https://github.com/TonyStark042/Reflex.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。