从正在学习的智能体轨迹中逆向推断其奖励函数，突破传统静态假设。
Inverse Reinforcement Learning from Non-Stationary Learning Agents
- 用动态行为克隆法捕捉学习过程中变化的策略分布
- 仅需少量不同时期轨迹即能准确估计奖励函数
- 适用于研究在线学习智能体的内在动机，如强化学习研究者
本文研究一种逆强化学习问题：利用学习智能体在训练过程中生成的轨迹数据，推断其奖励函数。为此，我们提出一种新方法，通过估计智能体的策略参数来还原其奖励函数。该方法基于改进的行为克隆算法——捆绑行为克隆（bundle behavior cloning），利用学习过程中不同时间点生成的少量轨迹，学习一组匹配观测动作分布的策略。随后，使用这些克隆策略训练神经网络以估计智能体的奖励函数。我们提供了理论分析，证明该方法在复杂度边界上优于标准行为克隆；数值实验也在一个强化学习任务中验证了方法的有效性。
原文摘要 · Abstract (English)
In this paper, we study an inverse reinforcement learning problem that involves learning the reward function of a learning agent using trajectory data collected while this agent is learning its optimal policy. To address this problem, we propose an inverse reinforcement learning method that allows us to estimate the policy parameters of the learning agent which can then be used to estimate its reward function. Our method relies on a new variant of the behavior cloning algorithm, which we call bundle behavior cloning, and uses a small number of trajectories generated by the learning agent's policy at different points in time to learn a set of policies that match the distribution of actions observed in the sampled trajectories. We then use the cloned policies to train a neural network model that estimates the reward function of the learning agent. We provide a theoretical analysis to show a complexity result on bound guarantees for our method that beats standard behavior cloning as well as numerical experiments for a reinforcement learning problem that validate the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。