同时预训练策略和价值网络，显著提升强化学习样本效率。
Actor-Critic Pretraining for Proximal Policy Optimization
- 用专家数据预训练策略网络，用策略回放结果预训练价值网络。
- 在15个机器人任务上平均节省86.1%环境交互次数。
- 适合需要高效训练的机器人控制场景。
强化学习中的演员-评论家算法虽能实现自主学习，但通常需大量环境交互，限制了其在机器人领域的应用。利用专家数据可减少所需交互次数。常见做法是仅对策略网络进行行为克隆预训练，随后通过强化学习微调。然而，评论家网络的初始化却很少受到关注，尽管其在策略优化中起关键作用。本文提出一种针对近端策略优化（PPO）等演员-评论家算法的预训练方法，使用专家演示数据同时初始化策略和评论家网络：策略网络通过行为克隆预训练，评论家网络则基于预训练策略的回放轨迹获得回报进行预训练。该方法在15个模拟机器人操控与行走任务上进行了评估。实验结果表明，相比无预训练，该方法平均提升样本效率86.1%；相比仅策略预训练，提升30.9%。
原文摘要 · Abstract (English)
Reinforcement learning (RL) actor-critic algorithms enable autonomous learning but often require a large number of environment interactions, which limits their applicability in robotics. Leveraging expert data can reduce the number of required environment interactions. A common approach is actor pretraining, where the actor network is initialized via behavioral cloning on expert demonstrations and subsequently fine-tuned with RL. In contrast, the initialization of the critic network has received little attention, despite its central role in policy optimization. This paper proposes a pretraining approach for actor-critic algorithms like Proximal Policy Optimization (PPO) that uses expert demonstrations to initialize both networks. The actor is pretrained via behavioral cloning, while the critic is pretrained using returns obtained from rollouts of the pretrained policy. The approach is evaluated on 15 simulated robotic manipulation and locomotion tasks. Experimental results show that actor-critic pretraining improves sample efficiency by 86.1% on average compared to no pretraining and by 30.9% to actor-only pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。