用快捷生成模型提升离线强化学习的训练与推理效率。
Scaling Offline RL via Efficient and Expressive Shortcut Models
- 引入快捷模型实现单阶段高效训练，替代传统迭代采样。
- 测试时通过学习的Q函数验证，支持串行与并行推理扩展。
- 在多个离线强化学习任务中表现优异，且计算资源越多效果越好。
扩散模型和流模型作为强大的生成方法，能够建模多样且多模态的行为分布。然而，由于其噪声采样的迭代特性，在离线强化学习（Offline RL）中应用仍面临挑战。本文提出可扩展离线强化学习（SORL），一种利用新型生成模型——快捷模型的新算法，实现训练与推理的高效扩展。SORL 的策略能捕捉复杂数据分布，并通过单阶段训练流程简单高效地完成优化。测试时，借助学习到的Q函数作为验证器，实现串行与并行推理的可扩展性。实验表明，SORL 在多种离线强化学习任务中均取得优异性能，并表现出随着测试时计算资源增加而持续提升的良好扩展性。代码已开源：nico-espinosadice.github.io/projects/sorl。
原文摘要 · Abstract (English)
Diffusion and flow models have emerged as powerful generative approaches capable of modeling diverse and multimodal behavior. However, applying these models to offline reinforcement learning (RL) remains challenging due to the iterative nature of their noise sampling processes, making policy optimization difficult. In this paper, we introduce Scalable Offline Reinforcement Learning (SORL), a new offline RL algorithm that leverages shortcut models - a novel class of generative models - to scale both training and inference. SORL's policy can capture complex data distributions and can be trained simply and efficiently in a one-stage training procedure. At test time, SORL introduces both sequential and parallel inference scaling by using the learned Q-function as a verifier. We demonstrate that SORL achieves strong performance across a range of offline RL tasks and exhibits positive scaling behavior with increased test-time compute. We release the code at nico-espinosadice.github.io/projects/sorl.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。