提出新微调算法,避免智能体在在线学习时性能下降。
Fine-Tuning without Performance Degradation
- 基于在线表现估计逐步放开探索,减少初始阶段的性能波动。
- 实验显示该方法比现有算法更快收敛且性能下降更少。
- 适合需要稳定在线学习的强化学习应用,如机器人控制。
离线学习策略的在线微调仍是应用中的重大挑战。通常在微调初期,智能体因探索行为会严重破坏已学策略,导致性能下降。我们发现,许多离线到在线的算法在微调过程中要么出现性能下降,要么学习速度缓慢(甚至无改善)。为此,提出一种基于「Jump Start」的新微调算法,根据在线性能估计逐步增加探索程度。实验证明,该方法在多种场景下实现了快速微调,并显著降低性能下降,优于现有同类算法。
原文摘要 · Abstract (English)
Fine-tuning policies learned offline remains a major challenge in application domains. Monotonic performance improvement during \emph{fine-tuning} is often challenging, as agents typically experience performance degradation at the early fine-tuning stage. The community has identified multiple difficulties in fine-tuning a learned network online, however, the majority of progress has focused on improving learning efficiency during fine-tuning. In practice, this comes at a serious cost during fine-tuning: initially, agent performance degrades as the agent explores and effectively overrides the policy learned offline. We show across a range of settings, many offline-to-online algorithms exhibit either (1) performance degradation or (2) slow learning (sometimes effectively no improvement) during fine-tuning. We introduce a new fine-tuning algorithm, based on an algorithm called Jump Start, that gradually allows more exploration based on online estimates of performance. Empirically, this approach achieves fast fine-tuning and significantly reduces performance degradations compared with existing algorithms designed to do the same.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。