不靠对抗训练，直接优化策略匹配专家行为特征。
Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching
- 用状态转移特征与奖励向量的内积建模回报，直接优化策略差距。
- 仅需单次示范即可学习，在无动作标签时仍有效。
- 兼容现有强化学习算法，适合数据稀缺的控制任务。
逆强化学习（IRL）中，智能体通过与环境交互来模仿专家示范。传统方法将IRL视为对抗博弈：对手搜索奖励模型，学习者通过重复强化学习过程优化奖励。该方法计算成本高且难以稳定。本文提出一种新方法：通过显式策略优化，利用回报的线性分解——即后继特征与奖励向量的内积，设计基于策略梯度的IRL算法，直接最小化学习者与专家特征间的差距。所提非对抗方法无需学习显式奖励函数，可无缝集成至现有演员-评论家强化学习框架。显著地，该方法在仅提供状态信息而无专家动作标签的设置下仍有效，而行为克隆（BC）无法处理此场景。实验表明，该方法仅需一次专家示范即可学习，并在多种控制任务上取得更优性能。
原文摘要 · Abstract (English)
In inverse reinforcement learning (IRL), an agent seeks to replicate expert demonstrations through interactions with the environment. Traditionally, IRL is treated as an adversarial game, where an adversary searches over reward models, and a learner optimizes the reward through repeated RL procedures. This game-solving approach is both computationally expensive and difficult to stabilize. In this work, we propose a novel approach to IRL by direct policy optimization: exploiting a linear factorization of the return as the inner product of successor features and a reward vector, we design an IRL algorithm by policy gradient descent on the gap between the learner and expert features. Our non-adversarial method does not require learning a reward function and can be solved seamlessly with existing actor-critic RL algorithms. Remarkably, our approach works in state-only settings without expert action labels, a setting which behavior cloning (BC) cannot solve. Empirical results demonstrate that our method learns from as few as a single expert demonstration and achieves improved performance on various control tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。