用冻结的离线模型指导在线强化学习，无需保存离线数据集。
SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance
- 用状态-动作自适应系数融合离线与在线批评者网络。
- 在D4RL基准上超越现有最优离线转在线算法。
- 适合希望减少离线数据依赖的强化学习研究者。
离线转在线强化学习（O2O RL）先在离线数据上预训练模型，再通过在线微调优化策略。然而，现有方法通常需持续维护离线数据集以缓解分布外（OOD）数据影响，严重限制了在线样本的利用效率。为此，本文提出一种新范式——状态-动作条件化离线模型引导（SAMG）。该方法冻结预训练的离线批评者，为每个状态-动作样本提供紧凑的离线理解，从而无需重新训练于离线数据。冻结的离线批评者与在线目标批评者结合，权重由状态-动作自适应系数决定，该系数在训练中动态更新，用于捕捉样本的离线程度。实践中，SAMG可轻松集成至基于Q函数的算法。理论分析表明其具备良好最优性与更低估计误差。实验结果显示，SAMG在D4RL基准上优于当前最先进O2O RL算法。
原文摘要 · Abstract (English)
Offline-to-online (O2O) reinforcement learning (RL) pre-trains models on offline data and refines policies through online fine-tuning. However, existing O2O RL algorithms typically require maintaining the tedious offline datasets to mitigate the effects of out-of-distribution (OOD) data, which significantly limits their efficiency in exploiting online samples. To address this deficiency, we introduce a new paradigm for O2O RL called State-Action-Conditional Offline \Model Guidance (SAMG). It freezes the pre-trained offline critic to provide compact offline understanding for each state-action sample, thus eliminating the need for retraining on offline data. The frozen offline critic is incorporated with the online target critic weighted by a state-action-adaptive coefficient. This coefficient aims to capture the offline degree of samples at the state-action level, and is updated adaptively during training. In practice, SAMG could be easily integrated with Q-function-based algorithms. Theoretical analysis shows good optimality and lower estimation error. Empirically, SAMG outperforms state-of-the-art O2O RL algorithms on the D4RL benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。