让受限机器人模仿专家动作,通过动态对齐提升学习效果。
Action-Constrained Imitation Learning
- 用模型预测控制对齐专家轨迹,生成符合约束的替代数据集
- 在多个机器人任务中显著提升性能,样本效率优于基准方法
- 适合需要安全动作约束的机器人控制与资源分配场景
在机器人控制和资源分配等应用中,带动作约束的策略学习对确保安全行为至关重要。本文提出新的问题设定——动作约束模仿学习(ACIL),即受限模仿者需从具有更大动作空间的专家示范中学习。其核心挑战在于动作约束导致专家与模仿者状态占用度不匹配。为此,我们提出基于动态时间规整(DTW)距离的轨迹对齐方法DTWIL,将原始专家示范替换为遵循相似状态轨迹但满足动作约束的代理数据集。具体地,将轨迹对齐建模为规划问题,并通过模型预测控制求解。大量实验表明,使用DTWIL生成的数据集能显著提升多种机器人控制任务的性能,且在样本效率上优于多种基准模仿学习算法。代码已公开于https://github.com/NYCU-RL-Bandits-Lab/ACRL-Baselines。
原文摘要 · Abstract (English)
Policy learning under action constraints plays a central role in ensuring safe behaviors in various robot control and resource allocation applications. In this paper, we study a new problem setting termed Action-Constrained Imitation Learning (ACIL), where an action-constrained imitator aims to learn from a demonstrative expert with larger action space. The fundamental challenge of ACIL lies in the unavoidable mismatch of occupancy measure between the expert and the imitator caused by the action constraints. We tackle this mismatch through \textit{trajectory alignment} and propose DTWIL, which replaces the original expert demonstrations with a surrogate dataset that follows similar state trajectories while adhering to the action constraints. Specifically, we recast trajectory alignment as a planning problem and solve it via Model Predictive Control, which aligns the surrogate trajectories with the expert trajectories based on the Dynamic Time Warping (DTW) distance. Through extensive experiments, we demonstrate that learning from the dataset generated by DTWIL significantly enhances performance across multiple robot control tasks and outperforms various benchmark imitation learning algorithms in terms of sample efficiency. Our code is publicly available at https://github.com/NYCU-RL-Bandits-Lab/ACRL-Baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。