用进度约束提升行为树与强化学习结合的稳定性。
Progress Constraints for Reinforcement Learning in Behavior Trees
- 引入可行性估计器限制动作集，防止控制器互相抵消
- 在2D和仓库环境中实现更高样本效率和更强约束满足
- 适合需要安全决策与长期目标的强化学习应用
行为树（BTs）提供结构化且响应式的决策框架，常用于根据环境条件切换子控制器。强化学习（RL）虽能学习近似最优控制器，但面临稀疏奖励、安全探索和长时程信用分配难题。将BT与RL结合可互为补充：BT的设计蕴含结构化领域知识，简化RL训练；而RL可自动学习BT中的控制器。然而，简单集成可能导致某些控制器抵消其他控制器，破坏已达成的子目标，从而降低整体性能。为此，我们提出进度约束机制，基于理论上的行为树收敛性结果，利用可行性估计器限制允许的动作集合。在2D概念验证环境和高保真仓库环境中的实证评估表明，相比先前的BT-RL集成方法，该方法显著提升了性能、样本效率和约束满足度。
原文摘要 · Abstract (English)
Behavior Trees (BTs) provide a structured and reactive framework for decision-making, commonly used to switch between sub-controllers based on environmental conditions. Reinforcement Learning (RL), on the other hand, can learn near-optimal controllers but sometimes struggles with sparse rewards, safe exploration, and long-horizon credit assignment. Combining BTs with RL has the potential for mutual benefit: a BT design encodes structured domain knowledge that can simplify RL training, while RL enables automatic learning of the controllers within BTs. However, naive integration of BTs and RL can lead to some controllers counteracting other controllers, possibly undoing previously achieved subgoals, thereby degrading the overall performance. To address this, we propose progress constraints, a novel mechanism where feasibility estimators constrain the allowed action set based on theoretical BT convergence results. Empirical evaluations in a 2D proof-of-concept and a high-fidelity warehouse environment demonstrate improved performance, sample efficiency, and constraint satisfaction, compared to prior methods of BT-RL integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。