arXiv:2506.21427cs.LGcs.RO2025-06被引 9

用单步生成替代迭代采样,让生成式策略又快又稳。

Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning

  • 设计单步完成向量预测,跳过传统生成模型的多步采样过程。
  • 在离线与在线强化学习中速度提升数倍,性能媲美扩散模型。
  • 适合追求高效且表达能力强的决策系统研发者使用。

生成模型如扩散模型和流匹配能捕捉丰富的多模态动作分布,提升离线强化学习的策略表达能力,但其迭代采样带来高推理成本和训练不稳定性。本文提出单步完成策略(SSCP),通过增强的流匹配目标,直接从中间流样本预测完整动作向量,实现一次生成即可输出准确动作。在离策略演员-评论家框架中,SSCP兼具生成模型的表达力与单峰策略的训练和推理效率,无需长距离反向传播。该方法可有效扩展至离线、离线到在线及在线强化学习场景,在速度和适应性上显著优于基于扩散模型的基线。进一步将SSCP拓展至目标条件强化学习,使扁平策略无需显式分层推断即可利用子目标结构。SSCP在标准离线强化学习与行为克隆基准上表现优异,是一个通用、表达性强且高效的深度强化学习框架。代码已开源:https://github.com/PrajwalKoirala/SSCP-Single-Step-Completion-Policy。

原文摘要 · Abstract (English)

Generative models such as diffusion and flow-matching offer expressive policies for offline reinforcement learning (RL) by capturing rich, multimodal action distributions, but their iterative sampling introduces high inference costs and training instability due to gradient propagation across sampling steps. We propose the Single-Step Completion Policy (SSCP), a generative policy trained with an augmented flow-matching objective to predict direct completion vectors from intermediate flow samples, enabling accurate, one-shot action generation. In an off-policy actor-critic framework, SSCP combines the expressiveness of generative models with the training and inference efficiency of unimodal policies, without requiring long backpropagation chains. Our method scales effectively to offline, offline-to-online, and online RL settings, offering substantial gains in speed and adaptability over diffusion-based baselines. We further extend SSCP to goal-conditioned RL, enabling flat policies to exploit subgoal structures without explicit hierarchical inference. SSCP achieves strong results across standard offline RL and behavior cloning benchmarks, positioning it as a versatile, expressive, and efficient framework for deep RL and sequential decision-making. The code is available at https://github.com/PrajwalKoirala/SSCP-Single-Step-Completion-Policy.

强化学习生成模型单步生成高效策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。