arXiv:2506.09176cs.AIcs.LG2025-06ICML被引 6

让机器人主动判断何时要人帮忙,减少人类干预负担。

Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism

  • 用代理Q函数判断机器人与专家动作的匹配度,决定是否请求帮助。
  • 相比基线方法,人类介入成本降低40%,学习效率提升显著。
  • 能自动识别危险状态并获取高质量示范,适合需要安全控制的场景。

交互式模仿学习(IIL)使智能体通过人类干预习得目标行为,但现有方法对人类监督者认知负荷较高。本文提出自适应干预机制(AIM),一种新型机器人门控的IIL算法,可学习自适应的人类示范请求准则。AIM利用代理Q函数模拟人类干预规则,并根据智能体与人类动作的一致性动态调整干预请求。当智能体偏离专家行为时赋予高Q值,随着其能力提升则逐步降低,从而实现对专家对齐程度的实时评估,并在必要时请求协助。专家在环实验表明,该方法在连续与离散控制任务中均显著降低专家监控负担。相较于基于不确定性的基线方法Thrifty-DAgger,本方法在人类介入成本和学习效率上提升40%。此外,AIM能有效识别安全关键状态,收集更高品质的专家示范,减少整体所需专家数据量与环境交互次数。代码与演示视频见https://github.com/metadriverse/AIM。

原文摘要 · Abstract (English)

Interactive Imitation Learning (IIL) allows agents to acquire desired behaviors through human interventions, but current methods impose high cognitive demands on human supervisors. We propose the Adaptive Intervention Mechanism (AIM), a novel robot-gated IIL algorithm that learns an adaptive criterion for requesting human demonstrations. AIM utilizes a proxy Q-function to mimic the human intervention rule and adjusts intervention requests based on the alignment between agent and human actions. By assigning high Q-values when the agent deviates from the expert and decreasing these values as the agent becomes proficient, the proxy Q-function enables the agent to assess the real-time alignment with the expert and request assistance when needed. Our expert-in-the-loop experiments reveal that AIM significantly reduces expert monitoring efforts in both continuous and discrete control tasks. Compared to the uncertainty-based baseline Thrifty-DAgger, our method achieves a 40% improvement in terms of human take-over cost and learning efficiency. Furthermore, AIM effectively identifies safety-critical states for expert assistance, thereby collecting higher-quality expert demonstrations and reducing overall expert data and environment interactions needed. Code and demo video are available at https://github.com/metadriverse/AIM.

模仿学习人机协作自适应干预强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。