放弃预训练评判器，让模型在线学习更高效。
Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning
- 不依赖旧评判器，用全新评判器适应在线环境。
- 在多个任务上表现优于传统方法，挑战性任务提升显著。
- 适配主流算法，适合追求高效在线强化学习的开发者。
离线到在线强化学习旨在利用静态数据集预训练的策略，并通过在线交互进行优化。然而，直接复用离线训练的评判器会阻碍在线微调：随着策略和数据分布快速变化，继承的值估计可能与在线环境错位，导致策略改进不准确和探索效率低下。为此，我们提出无评判器预训练（Critic-Free Pretraining, CFP），彻底摒弃离线评判器训练，使全新初始化的评判器能无偏地适应在线环境。CFP 兼容多种主流 O2O 算法，在多样任务中表现一致或超越传统方法，尤其在若干高难度任务上取得显著提升。
原文摘要 · Abstract (English)
Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。