让强化学习模型适应近似对称性，提升泛化与抗噪能力。
Approximate Equivariance in Reinforcement Learning
- 用松弛群与可导向卷积设计近似等变网络
- 在近似对称任务中表现优于精确等变网络
- 适合有噪声或非完美对称性的连续控制与金融场景
等变神经网络在强化学习中表现出色，能提升样本效率和泛化能力，但仅在任务具有精确对称性时适用。许多实际问题中仅有近似对称性，强制施加精确对称性反而有害。本文提出适用于强化学习的近似等变方法，定义了近似等变马尔可夫决策过程，并理论分析其对最优 $Q$ 函数的影响。设计新型强化学习架构，采用松弛群与可导向卷积，在多个连续控制任务及使用真实金融数据的股票交易场景中进行实验。结果表明：当存在精确对称性时，近似等变网络性能与精确等变网络相当；而在近似对称场景中则显著更优。此外，该方法还提升了测试时对噪声的鲁棒性。代码已开源。
原文摘要 · Abstract (English)
Equivariant neural networks have shown great success in reinforcement learning, improving sample efficiency and generalization when there is symmetry in the task. However, in many problems, only approximate symmetry is present, which makes imposing exact symmetry inappropriate. Recently, approximately equivariant networks have been proposed for supervised classification and modeling physical systems. In this work, we develop approximately equivariant algorithms in reinforcement learning (RL). We define approximately equivariant MDPs and theoretically characterize the effect of approximate equivariance on the optimal $Q$ function. We propose novel RL architectures using relaxed group and steerable convolutions and experiment on several continuous control domains and stock trading with real financial data. Our results demonstrate that the approximately equivariant network performs on par with exactly equivariant networks when exact symmetries are present, and outperforms them when the domains exhibit approximate symmetry. As an added byproduct of these techniques, we observe increased robustness to noise at test time. Our code is available at https://github.com/jypark0/approx_equiv_rl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。