用AI发现奖励函数,让模仿学习更稳定高效。
On Discovering Algorithms for Adversarial Imitation Learning
- 用大模型驱动的进化算法自动搜索最优奖励分配函数。
- 在多个新环境中表现超越人工设计的基线方法。
- 首次实现可迁移的自适应奖励机制,适合强化学习研究者。
对抗性模仿学习(AIL)虽在专家示范有限时有效,但常因不稳定性受质疑。其通常分为密度比(DR)估计和奖励分配(RA)两部分:前者由判别器估计策略与专家在状态动作对上的相对访问频率;后者将该比率转化为用于训练策略的奖励信号。尽管密度估计已获大量研究,奖励分配对训练动态与最终性能的影响却长期被忽视。现有RA函数多源于人类设计的损失最小化目标。本文提出新思路:基于模仿策略的表现,数据驱动地发现RA函数。我们采用大模型引导的进化框架,高效探索RA函数空间,提出首个元学习的对抗性模仿学习算法——DAIL。令人惊讶的是,DAIL在未见过的环境和优化算法下均表现优异,超越当前最先进的人工设计基线。最后,我们分析了其提升训练稳定性的原因,揭示了RA函数在AIL稳定性中的关键作用。代码公开:https://github.com/shshnkreddy/DAIL。
原文摘要 · Abstract (English)
Adversarial Imitation Learning (AIL) methods, while effective in settings with limited expert demonstrations, are often considered unstable. These approaches typically decompose into two components: Density Ratio (DR) estimation $\frac{ρ_E}{ρ_π}$, where a discriminator estimates the relative occupancy of state-action pairs under the policy versus the expert; and Reward Assignment (RA), where this ratio is transformed into a reward signal used to train the policy. While significant research has focused on improving density estimation, the role of reward assignment in influencing training dynamics and final policy performance has been largely overlooked. RA functions in AIL are typically derived from divergence minimization objectives, relying heavily on human design and ingenuity. In this work, we take a different approach: we investigate the discovery of data-driven RA functions, i.e, based directly on the performance of the resulting imitation policy. To this end, we leverage an LLM-guided evolutionary framework that efficiently explores the space of RA functions, yielding \emph{Discovered Adversarial Imitation Learning} (DAIL), the first meta-learnt AIL algorithm. Remarkably, DAIL generalises across unseen environments and policy optimization algorithms, outperforming the current state-of-the-art of \emph{human-designed} baselines. Finally, we analyse why DAIL leads to more stable training, offering novel insights into the role of RA functions in the stability of AIL. Code is publicly available: https://github.com/shshnkreddy/DAIL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。