arXiv:2608.20208cs.CV2026-08

用自回归归一化流提升机器人操作的离线强化学习,大幅降低推理延迟。

RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

论文配图:RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation
图 1 · 摘自论文原文
  • 不用采样计算优势加权似然,加速策略优化
  • 将序列策略蒸馏为单步生成器,部署时低延迟
  • 在仿真和真实机器人上均表现优异,适合工业部署

离线强化学习利用预先收集的数据训练机器人策略,无需额外环境交互。然而,主流的扩散和流匹配策略缺乏可计算的似然性,限制了其在基于似然的离线RL后处理中的应用。自回归归一化流(AR-NF)兼具强大的动作建模能力和精确的似然评估,但其序列采样在策略优化和部署阶段带来显著计算开销。本文提出RoMAN-Flow(基于自回归归一化流的机器人操作),通过在两个阶段解决采样瓶颈,使AR-NF策略在机器人操作中具备实用性。在策略优化阶段,采用无采样的优势加权似然目标,直接赋予高优势动作更高似然值;在部署阶段,将优化后的自回归策略蒸馏为单步动作生成器,实现低延迟动作预测。在多个模拟操作基准和真实机器人平台上验证,RoMAN-Flow在保持竞争力策略性能的同时,显著降低推理延迟。代码已开源:https://github.com/konnyaku28/RoMAN-Flow。

原文摘要 · Abstract (English)

Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across multiple simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency. Code is available at https://github.com/konnyaku28/RoMAN-Flow.

离线强化学习自回归流机器人操作低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。