arXiv:2502.07645cs.RO2025-02被引 4

用动作集合替代单一动作标签,让模仿学习更抗错。

From Action Labels to Sets: Rethinking Action Supervision for Imitation Learning from Corrective Feedback

  • 用人类纠正构建动作集合,而非单个动作目标
  • 在噪声、相对和部分反馈下性能显著优于传统方法
  • 适合需要高鲁棒性的机器人操作场景

行为克隆(BC)将人类示范视为点式动作标签来优化策略。当动作标签准确时有效,但实际中人类提供动作常有误差,若将每个标签当作精确目标,会误导策略偏离真实期望行为,尤其在使用表达能力强的模型(如能量模型)时更为明显。为此,我们提出一种人机协作的新范式,将点式监督替换为集合式动作目标。提出对比性交互修正策略学习(CLIC),利用人类纠正构建并迭代优化理想动作集合,使策略在这些集合上分配概率质量,而非单一动作。该方法天然支持绝对与相对修正,可表示复杂多模态行为。大量仿真与真实机器人实验表明,该方法在准确数据下保持与当前最优相当,在噪声、相对及部分反馈下显著更鲁棒。代码已开源:https://clic-webpage.github.io/

原文摘要 · Abstract (English)

Behavior cloning (BC) optimizes policies by treating human demonstrations as pointwise action labels. While effective with accurate action labels, this formulation is brittle in practice: when human-provided actions are imperfect, treating each label as an exact target can steer the policy away from the underlying desired behavior, particularly when expressive models are used (e.g., energy-based models). As a result, we propose a human-in-the-loop alternative that replaces pointwise supervision with set-valued action targets. We introduce Contrastive policy Learning from Interactive Corrections (CLIC). CLIC leverages human corrections to construct and refine sets of desired actions, and optimizes a policy to place probability mass over these sets rather than over a single action target. This formulation naturally accommodates both absolute and relative corrections and can represent complex multi-modal behaviors. Extensive simulation and real-robot experiments show that the proposed approach leads to effective policy learning across diverse settings: CLIC remains competitive with the state of the art under accurate data while being substantially more robust under noisy, relative, and partial feedback. Our implementation is publicly available at https://clic-webpage.github.io/.

模仿学习人机交互鲁棒性动作集合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。