通过缩短轨迹长度，显著提升离线强化学习的可扩展性
Horizon Reduction Makes RL Scalable
- 提出轨迹截断技术，降低长时序任务对算法的干扰
- 在1000倍更大数据集上验证，传统方法性能饱和，新方法持续提升
- 适合研究大规模离线强化学习或需高效训练的工程应用
本文研究离线强化学习（offline RL）算法的可扩展性。理想的可扩展算法应在充足数据、算力和模型容量下解决任意复杂任务。我们使用最大达典型数据集1000倍规模的多样化数据集，在此前未被解决的挑战性任务上评估现有算法。结果发现，尽管数据量大幅增加，多数现有算法性能仍提前饱和，无法持续提升。我们假设长期限是导致其扩展性差的核心原因，并通过多组实验验证：长时序确实构成根本障碍。进一步证明多种轨迹截断技术能显著改善可扩展性。基于此，提出名为SHARSA的最小但高效的方法，显式压缩轨迹长度。SHARSA在所有测试方法中表现最佳，达到最优渐近性能与扩展行为，表明主动减少轨迹长度可释放离线RL的可扩展潜力。
原文摘要 · Abstract (English)
In this work, we study the scalability of offline reinforcement learning (RL) algorithms. In principle, a truly scalable offline RL algorithm should be able to solve any given problem, regardless of its complexity, given sufficient data, compute, and model capacity. We investigate if and how current offline RL algorithms match up to this promise on diverse, challenging, previously unsolved tasks, using datasets up to 1000x larger than typical offline RL datasets. We observe that despite scaling up data, many existing offline RL algorithms exhibit poor scaling behavior, saturating well below the maximum performance. We hypothesize that the horizon is the main cause behind the poor scaling of offline RL. We empirically verify this hypothesis through several analysis experiments, showing that long horizons indeed present a fundamental barrier to scaling up offline RL. We then show that various horizon reduction techniques substantially enhance scalability on challenging tasks. Based on our insights, we also introduce a minimal yet scalable method named SHARSA that effectively reduces the horizon. SHARSA achieves the best asymptotic performance and scaling behavior among our evaluation methods, showing that explicitly reducing the horizon unlocks the scalability of offline RL. Code: https://github.com/seohongpark/horizon-reduction
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。