在线学习专家轨迹的奖励函数，边看边更新。
In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminates
- 在轨迹进行中实时学习奖励函数，无需等完整轨迹结束。
- 算法实现次线性局部后悔，理论证明优于传统方法。
- 适合需要快速适应新任务的机器人控制场景。
逆强化学习（IRL）旨在从专家示范轨迹中学习最优奖励函数及对应策略。然而现有方法必须等待完整轨迹收集后才能开始学习，无法在轨迹进行中增量更新。本文提出一种在线双层优化框架：上层利用元正则化项动态调整奖励函数，下层学习对应策略，随着新状态-动作对不断观测持续更新。提出的新算法保证了次线性局部后悔 $O( ext{sqrt}{T} + ext{log} T + ext{sqrt}{T} ext{log} T)$。若奖励函数为线性，则后悔率可降至 $O( ext{log} T)$。实验验证了算法有效性。
原文摘要 · Abstract (English)
Inverse reinforcement learning (IRL) aims to learn a reward function and a corresponding policy that best fit the demonstrated trajectories of an expert. However, current IRL works cannot learn incrementally from an ongoing trajectory because they have to wait to collect at least one complete trajectory to learn. To bridge the gap, this paper considers the problem of learning a reward function and a corresponding policy while observing the initial state-action pair of an ongoing trajectory and keeping updating the learned reward and policy when new state-action pairs of the ongoing trajectory are observed. We formulate this problem as an online bi-level optimization problem where the upper level dynamically adjusts the learned reward according to the newly observed state-action pairs with the help of a meta-regularization term, and the lower level learns the corresponding policy. We propose a novel algorithm to solve this problem and guarantee that the algorithm achieves sub-linear local regret $O(\sqrt{T}+\log T+\sqrt{T}\log T)$. If the reward function is linear, we prove that the proposed algorithm achieves sub-linear regret $O(\log T)$. Experiments are used to validate the proposed algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。