用离策略方法提升模仿学习效率，减少样本需求。
Enabling Off-Policy Imitation Learning with Deep Actor Critic Stabilization
- 结合离策略框架与双Q网络稳定机制
- 仅需1/4样本即可达到专家水平表现
- 适合需要高效训练的机器人控制场景
强化学习在训练复杂策略时常因不稳定和收敛慢而受阻，尤其在奖励设计困难的情况下。模仿学习通过专家示范可避免对奖励函数的依赖，但当前主流方法（如GAIL）因基于on-policy算法（如TRPO），存在严重样本低效问题。本文提出一种对抗性模仿学习算法，引入离策略学习以提升样本效率。通过结合离策略框架与双Q网络稳定技术，以及无需奖励函数推断的价值学习，实验表明该方法显著降低达到稳健专家行为所需样本量，验证了其有效性。
原文摘要 · Abstract (English)
Learning complex policies with Reinforcement Learning (RL) is often hindered by instability and slow convergence, a problem exacerbated by the difficulty of reward engineering. Imitation Learning (IL) from expert demonstrations bypasses this reliance on rewards. However, state-of-the-art IL methods, exemplified by Generative Adversarial Imitation Learning (GAIL)Ho et. al, suffer from severe sample inefficiency. This is a direct consequence of their foundational on-policy algorithms, such as TRPO Schulman et.al. In this work, we introduce an adversarial imitation learning algorithm that incorporates off-policy learning to improve sample efficiency. By combining an off-policy framework with auxiliary techniques specifically, in this case a double Q network based stabilization and value learning without reward function inference we demonstrate a reduction in the samples required to robustly match expert behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。