用动态权重融合专家与强化学习动作，显著提升训练效率。
Dynamic Action Interpolation: A Universal Approach for Accelerating Reinforcement Learning with Expert Guidance
- 通过时变权重动态混合专家与RL动作，无需额外网络或损失函数。
- 在MuJoCo上平均提升早期性能160%以上，最终性能超50%。
- 适合需要快速收敛的连续控制任务，尤其适用于新手研究者。
强化学习因样本效率低下而受限，尤其在训练初期需大量环境交互才能表现良好。现有方法常引入先验知识，但带来显著的架构与实现复杂性。本文提出动态动作插值（DAI），一种通用且简洁的框架，通过时变权重α(t)插值专家与强化学习动作，可无缝集成至任意演员-评论家算法，仅需几行代码，无需辅助网络或额外损失。理论分析表明，DAI重塑状态访问分布，加速价值函数学习并保持收敛性保证。在MuJoCo连续控制任务上的实证评估显示，DAI平均提升早期性能超过160%，最终性能提升逾50%；其中Humanoid任务早期表现提升4倍，收敛时提升2倍。这些结果挑战了复杂架构改造是实现高效强化学习的必要条件这一假设。
原文摘要 · Abstract (English)
Reinforcement learning (RL) suffers from severe sample inefficiency, especially during early training, requiring extensive environmental interactions to perform competently. Existing methods tend to solve this by incorporating prior knowledge, but introduce significant architectural and implementation complexity. We propose Dynamic Action Interpolation (DAI), a universal yet straightforward framework that interpolates expert and RL actions via a time-varying weight $α(t)$, integrating into any Actor-Critic algorithm with just a few lines of code and without auxiliary networks or additional losses. Our theoretical analysis shows that DAI reshapes state visitation distributions to accelerate value function learning while preserving convergence guarantees. Empirical evaluations across MuJoCo continuous control tasks demonstrate that DAI improves early-stage performance by over 160\% on average and final performance by more than 50\%, with the Humanoid task showing a 4$\times$ improvement early on and a 2$\times$ gain at convergence. These results challenge the assumption that complex architectural modifications are necessary for sample-efficient reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。