让奖励函数对齐任务目标，而非仅匹配数据。
Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
- 用专家示范作为弱监督，生成多个符合任务的奖励函数
- 通过对抗机制验证策略完成任务的能力，提升泛化性
- 适合复杂任务和迁移学习场景，效果优于传统方法
许多模仿学习（IL）算法使用逆强化学习（IRL）推断与示范数据对齐的奖励函数，但这些奖励函数常无法捕捉底层任务目标。本文提出一种新型基于IRL的模仿学习框架，强调任务对齐而非传统数据对齐。该框架为半监督方法，利用专家示范作为弱监督，生成一组与任务对齐而非仅与数据对齐的候选奖励函数；随后采用对抗机制训练策略，以集体验证其完成任务的能力。我们提供了该框架缓解任务-奖励错位的理论分析，并给出了实用实现。实验表明，在复杂环境和迁移学习场景中，本框架显著优于传统模仿学习基线。
原文摘要 · Abstract (English)
Many imitation learning (IL) algorithms use inverse reinforcement learning (IRL) to infer a reward function that aligns with the demonstration. However, the inferred reward functions often fail to capture the underlying task objectives. In this paper, we propose a novel framework for IRL-based IL that prioritizes task alignment over conventional data alignment. Our framework is a semi-supervised approach that leverages expert demonstrations as weak supervision to derive a set of candidate reward functions that align with the task rather than only with the data. It then adopts an adversarial mechanism to train a policy with this set of reward functions to gain a collective validation of the policy's ability to accomplish the task. We provide theoretical insights into this framework's ability to mitigate task-reward misalignment and present a practical implementation. Our experimental results show that our framework outperforms conventional IL baselines in complex and transfer learning scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。