arXiv:2605.11020cs.LGcs.AI2026-05中稿 · as a conference pa…

提出TRIRL算法,实现无需每轮求解强化学习的稳定奖励学习。

Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates

论文配图:Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
图 1 · 摘自论文原文
  • 基于信任域优化局部策略更新,显式求解对偶目标
  • 在多个任务上性能提升2.4倍,且奖励函数可泛化至动态变化
  • 兼顾稳定性与单调改进,适合需可靠奖励学习的场景

逆强化学习通常被建模为在匹配专家轨迹分布的前提下最大化熵。传统对偶上升方法虽保证单调性能提升,但每轮需完全求解强化学习问题以计算对偶梯度;而近期对抗方法通过直接优化原始问题并使用判别器提供奖励来避免该开销,却牺牲了稳定性与对偶单调性。本文通过理论洞察——对奖励函数更新的信任域最优策略,等价于同一方向更小更新下的全局最优策略——实现了不需每轮完全求解强化学习即可保持对偶单调改进。所提算法TRIRL(信任域逆强化学习)在多个挑战性任务中,聚合四分位均值表现优于现有模仿学习方法2.4倍,且恢复的奖励函数可泛化至系统动力学变化。

原文摘要 · Abstract (English)

Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. More recent adversarial methods avoid this cost at the expense of stability and monotonic dual improvement, by directly optimizing the primal problem and using a discriminator to provide rewards. In this work, we bridge the gap between these approaches by enabling monotonic improvement of the reward function and policy without having to fully solve an RL problem at every iteration. Our key theoretical insight is that a trust-region-optimal policy for a reward function update can be globally optimal for a smaller update in the same direction. This smaller update allows us to explicitly optimize the dual objective while only relying on a local search around the current policy. In doing so, our approach avoids the training instabilities of adversarial methods, offers monotonic performance improvement, and learns a reward function in the traditional sense of IRL--one that can be globally optimized to match expert demonstrations. Our proposed algorithm, Trust Region Inverse Reinforcement Learning (TRIRL), outperforms state-of-the-art imitation learning methods across multiple challenging tasks by a factor of 2.4x in terms of aggregate inter-quartile mean, while recovering reward functions that generalize to system dynamics shifts.

逆强化学习信任域模仿学习奖励泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。