用无分类器扩散生成提升离线到在线强化学习的数据质量
Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation
- 用无分类器引导扩散生成高质量数据，弥合离线与在线分布差距
- 在D4RL基准上使IQL、PEX等方法平均性能提升15%
- 可无缝集成现有算法，适合追求高效在线优化的研究者
离线到在线强化学习（O2O RL）旨在对预训练的离线策略进行在线微调，以减少昂贵的在线交互。现有方法利用离线数据集生成符合在线数据分布的数据用于增强，但生成数据仍存在与真实在线数据的分布差距，限制整体性能。为此，我们提出一种新的数据增强方法——无分类器扩散生成（CFDG）。该方法无需额外分类器训练开销，借助无分类器引导扩散模型，显著提升离线与在线数据分布差异下的生成质量。同时，采用重加权机制，使更多生成数据更贴近在线数据分布，在提升性能的同时保持智能体稳定性。实验表明，CFDG优于简单回放两类数据或使用标准扩散模型生成新数据。本方法具有通用性，可集成至现有O2O RL算法中。在主流算法IQL、PEX和APL中引入CFDG，于D4RL基准（如MuJoCo和AntMaze）上实现平均15%的性能提升。
原文摘要 · Abstract (English)
Offline-to-online Reinforcement Learning (O2O RL) aims to perform online fine-tuning on an offline pre-trained policy to minimize costly online interactions. Existing work used offline datasets to generate data that conform to the online data distribution for data augmentation. However, generated data still exhibits a gap with the online data, limiting overall performance. To address this, we propose a new data augmentation approach, Classifier-Free Diffusion Generation (CFDG). Without introducing additional classifier training overhead, CFDG leverages classifier-free guidance diffusion to significantly enhance the generation quality of offline and online data with different distributions. Additionally, it employs a reweighting method to enable more generated data to align with the online data, enhancing performance while maintaining the agent's stability. Experimental results show that CFDG outperforms replaying the two data types or using a standard diffusion model to generate new data. Our method is versatile and can be integrated with existing offline-to-online RL algorithms. By implementing CFDG to popular methods IQL, PEX and APL, we achieve a notable 15% average improvement in empirical performance on the D4RL benchmark such as MuJoCo and AntMaze.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。