用动作映射提升约束环境下强化学习的效率和收敛速度。
Action Mapping for Reinforcement Learning in Continuous Environments with Constraints
- 将可行动作学习与策略优化解耦，减少搜索空间。
- 在连续动作空间中显著提升训练性能，即使可行性模型不完美。
- 适合需要高效安全决策的机器人控制等场景。
深度强化学习（DRL）在多个领域取得成功，但在存在约束的环境中应用仍具挑战，主要源于样本效率低和收敛慢。现有研究尝试引入模型知识以缓解问题，尤其是利用评估动作可行性的模型。然而，在连续动作空间中高效集成此类可行性模型仍不简单。本文提出一种新的DRL训练策略——动作映射，通过利用可行性模型来简化学习过程。该方法将可行动作的学习与策略优化分离，使DRL代理只需从缩减后的可行动作集中选择最优动作。实验表明，动作映射在具有连续动作空间的约束环境中显著提升了训练性能，尤其在可行性模型不完善的情况下表现优异。
原文摘要 · Abstract (English)
Deep reinforcement learning (DRL) has had success across various domains, but applying it to environments with constraints remains challenging due to poor sample efficiency and slow convergence. Recent literature explored incorporating model knowledge to mitigate these problems, particularly through the use of models that assess the feasibility of proposed actions. However, integrating feasibility models efficiently into DRL pipelines in environments with continuous action spaces is non-trivial. We propose a novel DRL training strategy utilizing action mapping that leverages feasibility models to streamline the learning process. By decoupling the learning of feasible actions from policy optimization, action mapping allows DRL agents to focus on selecting the optimal action from a reduced feasible action set. We demonstrate through experiments that action mapping significantly improves training performance in constrained environments with continuous action spaces, especially with imperfect feasibility models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。