让机器人从人类错误示范中学习，通过推测'本该怎么做'来提升模仿能力。
Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrations
- 引入反事实动作扩展数据集，推测人类本意而非死学错误行为。
- 在噪声演示下性能优于主流方法，跨用户与技能水平均有效。
- 适合真实场景中人类示范不完美的机器人学习任务。
从人类学习具有挑战性，因人类是不完美的教师。日常演示常包含噪声动作和次优行为（如目标过冲）。现有方法多直接匹配人类行为，但受限于原始数据的缺陷。本文提出反事实行为克隆（Counter-BC），通过假设所有示范背后存在一个理想策略，而噪声与次优性掩盖了真实意图。Counter-BC在人类示范附近构建反事实动作，自动调整示范以提炼出简洁、符合趋势的策略。理论上证明其可从不完美数据、多用户及不同技能水平的教师中提取相似且简单的策略。实验证明，在含噪声演示、标准数据集和真实人类教师场景中，相比先进基线，能更有效地学习。结果表明，考虑潜在意图而非原样模仿，可显著提升机器人学习效果。
原文摘要 · Abstract (English)
Learning from humans is challenging because people are imperfect teachers. When everyday humans show the robot a new task they want it to perform, humans inevitably make errors (e.g., inputting noisy actions) and provide suboptimal examples (e.g., overshooting the goal). Existing methods often learn by matching some or all of the human's behavior --- but this approach is fundamentally limited because the demonstrations themselves are imperfect. In this work we advance offline imitation learning by enabling robots to extrapolate across nearby actions, instead of only considering what the human actually showed. We achieve this by hypothesizing that all of the human's demonstrations are trying to convey an underlying policy, while the noise and sub-optimality within their behaviors obfuscates the data and introduces unintentional complexity. To recover the underlying policy and learn what the human teacher meant, we introduce Counter-BC, a generalized version of behavior cloning. Counter-BC expands the dataset to include actions close to behaviors the human demonstrated (i.e., counterfactual actions that the human teacher could have intended, but did not actually show). During training Counter-BC autonomously modifies the human's demonstrations within this expanded region to reach a simplified policy that explains the underlying trends in the human's dataset. Theoretically, we prove that Counter-BC can extract a simple and similar-to-demonstration policy from imperfect data, multiple users, and teachers of varying skill levels. Empirically, we compare Counter-BC to state-of-the-art alternatives in settings with noisy demonstrations, standardized datasets, and real human teachers. Overall, we find that trying to extrapolate what the human teacher meant by considering nearby actions can lead to more proficient learning from humans. See videos of our work here: https://youtu.be/XaeOZWhTt68
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。