用大模型解析指令，让机器人更准地理解任务重点。
Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Language
- 用语言指令生成状态相关性掩码，屏蔽无关信息
- 在仅用1/4数据下性能比之前方法高15%
- 能处理模糊指令，适合真实场景的机器人学习
机器人可通过演示学习奖励函数以适应用户偏好，但数据有限时，奖励模型易过拟合于虚假关联而无法泛化。这是因为演示只展示如何执行任务，未说明任务中何者重要，导致模型关注无关状态细节。自然语言可更直接指明关注重点，理论上能解决与演示一致的多个奖励函数之间的歧义。然而现有语言条件奖励学习方法通常将指令视为简单条件信号，未充分挖掘其消歧潜力。此外，真实指令本身常具模糊性，简单条件不可靠。本文核心洞察是：演示与语言携带互补信息——演示展示行为方式，语言指定重要性。我们提出掩码逆强化学习（Masked IRL），利用大语言模型（LLM）融合二者优势。该框架从语言指令推断状态相关性掩码，并强制对无关状态成分保持不变。当指令模糊时，通过LLM推理结合演示上下文澄清指令。在仿真和真实机器人上，相比以往语言条件逆强化学习方法，本方法最多减少4.7倍数据用量，性能提升达15%，验证了更高的样本效率、泛化能力及对模糊语言的鲁棒性。
原文摘要 · Abstract (English)
Robots can adapt to user preferences by learning reward functions from demonstrations, but with limited data, reward models often overfit to spurious correlations and fail to generalize. This happens because demonstrations show robots how to do a task but not what matters for that task, causing the model to focus on irrelevant state details. Natural language can more directly specify what the robot should focus on, and, in principle, disambiguate between many reward functions consistent with the demonstrations. However, existing language-conditioned reward learning methods typically treat instructions as simple conditioning signals, without fully exploiting their potential to resolve ambiguity. Moreover, real instructions are often ambiguous themselves, so naive conditioning is unreliable. Our key insight is that these two input types carry complementary information: demonstrations show how to act, while language specifies what is important. We propose Masked Inverse Reinforcement Learning (Masked IRL), a framework that uses large language models (LLMs) to combine the strengths of both input types. Masked IRL infers state-relevance masks from language instructions and enforces invariance to irrelevant state components. When instructions are ambiguous, it uses LLM reasoning to clarify them in the context of the demonstrations. In simulation and on a real robot, Masked IRL outperforms prior language-conditioned IRL methods by up to 15% while using up to 4.7 times less data, demonstrating improved sample-efficiency, generalization, and robustness to ambiguous language. Project page: https://MIT-CLEAR-Lab.github.io/Masked-IRL and Code: https://github.com/MIT-CLEAR-Lab/Masked-IRL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。