解决离线预训练模型在线微调时价值估计不准的问题，提升强化学习性能。
Online Pre-Training for Offline-to-Online Reinforcement Learning
- 引入在线预训练阶段，为在线微调定制新价值函数。
- 在D4RL多个环境中平均性能提升30%。
- 适合需要稳定在线学习的强化学习应用。
离线到在线强化学习旨在结合离线与在线强化学习的优势：先离线预训练智能体，再通过在线交互进行微调。然而，近期研究发现，由于分布偏移导致的价值估计不准确，离线预训练的智能体在在线微调中常表现不佳，甚至随机初始化反而更有效。本文提出一种新方法——在线预训练（OPT），专为解决离线预训练智能体在在线微调中价值估计不准的问题而设计。OPT引入一个新阶段：在线预训练，使价值函数能够专门针对有效的在线微调进行训练。在TD3和SPOT上的实现表明，OPT在包括MuJoCo、Antmaze和Adroit在内的广泛D4RL环境中，平均性能提升了30%。
原文摘要 · Abstract (English)
Offline-to-online reinforcement learning (RL) aims to integrate the complementary strengths of offline and online RL by pre-training an agent offline and subsequently fine-tuning it through online interactions. However, recent studies reveal that offline pre-trained agents often underperform during online fine-tuning due to inaccurate value estimation caused by distribution shift, with random initialization proving more effective in certain cases. In this work, we propose a novel method, Online Pre-Training for Offline-to-Online RL (OPT), explicitly designed to address the issue of inaccurate value estimation in offline pre-trained agents. OPT introduces a new learning phase, Online Pre-Training, which allows the training of a new value function tailored specifically for effective online fine-tuning. Implementation of OPT on TD3 and SPOT demonstrates an average 30% improvement in performance across a wide range of D4RL environments, including MuJoCo, Antmaze, and Adroit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。