arXiv:2505.00913cs.LGcs.AI2025-05被引 1

提出新微调算法,避免智能体在在线学习时性能下降。

Fine-Tuning without Performance Degradation

  • 基于在线表现估计逐步放开探索,减少初始阶段的性能波动。
  • 实验显示该方法比现有算法更快收敛且性能下降更少。
  • 适合需要稳定在线学习的强化学习应用,如机器人控制。

离线学习策略的在线微调仍是应用中的重大挑战。通常在微调初期,智能体因探索行为会严重破坏已学策略,导致性能下降。我们发现,许多离线到在线的算法在微调过程中要么出现性能下降,要么学习速度缓慢(甚至无改善)。为此,提出一种基于「Jump Start」的新微调算法,根据在线性能估计逐步增加探索程度。实验证明,该方法在多种场景下实现了快速微调,并显著降低性能下降,优于现有同类算法。

原文摘要 · Abstract (English)

Fine-tuning policies learned offline remains a major challenge in application domains. Monotonic performance improvement during \emph{fine-tuning} is often challenging, as agents typically experience performance degradation at the early fine-tuning stage. The community has identified multiple difficulties in fine-tuning a learned network online, however, the majority of progress has focused on improving learning efficiency during fine-tuning. In practice, this comes at a serious cost during fine-tuning: initially, agent performance degrades as the agent explores and effectively overrides the policy learned offline. We show across a range of settings, many offline-to-online algorithms exhibit either (1) performance degradation or (2) slow learning (sometimes effectively no improvement) during fine-tuning. We introduce a new fine-tuning algorithm, based on an algorithm called Jump Start, that gradually allows more exploration based on online estimates of performance. Empirically, this approach achieves fast fine-tuning and significantly reduces performance degradations compared with existing algorithms designed to do the same.

强化学习微调策略优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。