提出统一评估框架与精简代码,让离线强化学习更透明可比。
A Clean Slate for Offline Reinforcement Learning
- 构建清晰分类体系与在线调参预算量化协议
- 实现单文件极简实现,速度大幅提升
- 基于统一空间设计新算法,性能显著超越基线
离线强化学习进展受制于模糊的问题定义和混杂的算法设计,导致实现不一致、消融实验不足、评估不公平。尽管离线RL明确避免环境交互,但先前方法常依赖大量未记录的在线评估进行超参数调优,使方法比较复杂化。此外,现有参考实现间在样板代码上差异显著,掩盖了核心算法贡献。为此,我们首先引入严谨的分类体系与透明的评估协议,明确定量在线调参预算。为解决算法设计不透明问题,我们提供多种无模型与有模型离线RL方法的干净、极简、单文件实现,显著提升清晰度并实现大幅加速。基于这些简化实现,我们提出Unifloral,一个统一算法,将多种先前方法纳入单一综合超参数空间,支持共享超参数空间内的算法开发。使用Unifloral与严谨评估协议,我们开发出两种新算法——TD3-AWR(无模型)和MoBRAC(有模型),显著优于现有基线。实现已公开于https://github.com/EmptyJackson/unifloral。
原文摘要 · Abstract (English)
Progress in offline reinforcement learning (RL) has been impeded by ambiguous problem definitions and entangled algorithmic designs, resulting in inconsistent implementations, insufficient ablations, and unfair evaluations. Although offline RL explicitly avoids environment interaction, prior methods frequently employ extensive, undocumented online evaluation for hyperparameter tuning, complicating method comparisons. Moreover, existing reference implementations differ significantly in boilerplate code, obscuring their core algorithmic contributions. We address these challenges by first introducing a rigorous taxonomy and a transparent evaluation protocol that explicitly quantifies online tuning budgets. To resolve opaque algorithmic design, we provide clean, minimalistic, single-file implementations of various model-free and model-based offline RL methods, significantly enhancing clarity and achieving substantial speed-ups. Leveraging these streamlined implementations, we propose Unifloral, a unified algorithm that encapsulates diverse prior approaches within a single, comprehensive hyperparameter space, enabling algorithm development in a shared hyperparameter space. Using Unifloral with our rigorous evaluation protocol, we develop two novel algorithms - TD3-AWR (model-free) and MoBRAC (model-based) - which substantially outperform established baselines. Our implementation is publicly available at https://github.com/EmptyJackson/unifloral.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。