用乐观批判器提升模仿学习，减少一半训练次数。
IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic
- 交替更新成本与策略，多批评器估计不确定性并驱动探索
- 在表格设置下理论保证最优，实践中性能显著提升
- 适合需要高效模仿学习的强化学习研究者
本文提出SOAR框架用于模仿学习。该框架是一种算法模板，通过主对偶风格的交替更新成本与策略来从专家示范中学习策略。在策略更新中，利用多批评器估计批评器不确定性，并构建乐观批评器以驱动探索。在表格设置下实例化时，所得算法具有理论保证，性能达到当前最佳ε水平。实际应用中，将该模板应用于基于Soft Actor Critic的模仿学习算法（如f-IRL、ML-IRL和CSIL），在多个MuJoCo环境中均持续提升性能。总体而言，使用SOAR可使达成相同性能所需的训练轮次减少一半。
原文摘要 · Abstract (English)
This paper introduces the SOAR framework for imitation learning. SOAR is an algorithmic template that learns a policy from expert demonstrations with a primal dual style algorithm that alternates cost and policy updates. Within the policy updates, the SOAR framework uses an actor critic method with multiple critics to estimate the critic uncertainty and build an optimistic critic fundamental to drive exploration. When instantiated in the tabular setting, we get a provable algorithm with guarantees that matches the best known results in $ε$. Practically, the SOAR template is shown to boost consistently the performance of imitation learning algorithms based on Soft Actor Critic such as f-IRL, ML-IRL and CSIL in several MuJoCo environments. Overall, thanks to SOAR, the required number of episodes to achieve the same performance is reduced by half.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。