arXiv:2603.14245cs.LGcs.AI2026-03

通过引导生成起点和控制熵,让流模型政策更快更会探索。

GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies

  • 用条件变分自编码器构建状态相关的起始点,实现高效生成。
  • 引入熵正则化,使策略在确定性和探索性间动态平衡。
  • 在连续控制任务中显著优于现有方法,适合需要高效探索的RL场景。

流匹配策略在强化学习中展现出巨大潜力,能捕捉复杂多模态的动作分布。然而其实际应用常受限于高昂的推理延迟和低效的在线探索。尽管已有研究采用单步蒸馏加速推理,但初始噪声分布的结构仍被忽视,这带来显著未开发的潜力。这一被忽略的因素,加上对策略随机性的控制难题,是提升蒸馏流匹配策略的关键。为此,我们提出GoldenStart(GSFlow),一种结合Q引导先验和显式熵控制的策略蒸馏方法。不从无信息噪声开始生成,而是引入由条件变分自编码器建模的Q引导先验。该状态相关先验将单步生成的起始点定位到高Q区域,有效提供“黄金起点”,使策略快速抵达有前景的动作。此外,为实现有效在线探索,我们让蒸馏后的执行器输出随机分布而非确定性点,通过熵正则化控制,使策略可在纯利用与有原则的探索之间切换。实验表明,通过设计生成起始点并显式控制策略熵,可实现高效且具探索性的策略,弥合生成模型与实用的演员-评论家方法之间的鸿沟。我们在离线与在线连续控制基准上进行了广泛实验,结果表明该方法显著超越先前最先进方法。代码将发布于 https://github.com/ZhHe11/GSFlow-RL。

原文摘要 · Abstract (English)

Flow-matching policies hold great promise for reinforcement learning (RL) by capturing complex, multi-modal action distributions. However, their practical application is often hindered by prohibitive inference latency and ineffective online exploration. Although recent works have employed one-step distillation for fast inference, the structure of the initial noise distribution remains an overlooked factor that presents significant untapped potential. This overlooked factor, along with the challenge of controlling policy stochasticity, constitutes two critical areas for advancing distilled flow-matching policies. To overcome these limitations, we propose GoldenStart (GSFlow), a policy distillation method with Q-guided priors and explicit entropy control. Instead of initializing generation from uninformed noise, we introduce a Q-guided prior modeled by a conditional VAE. This state-conditioned prior repositions the starting points of the one-step generation process into high-Q regions, effectively providing a "golden start" that shortcuts the policy to promising actions. Furthermore, for effective online exploration, we enable our distilled actor to output a stochastic distribution instead of a deterministic point. This is governed by entropy regularization, allowing the policy to shift from pure exploitation to principled exploration. Our integrated framework demonstrates that by designing the generative startpoint and explicitly controlling policy entropy, it is possible to achieve efficient and exploratory policies, bridging the generative models and the practical actor-critic methods. We conduct extensive experiments on offline and online continuous control benchmarks, where our method significantly outperforms prior state-of-the-art approaches. Code will be available at https://github.com/ZhHe11/GSFlow-RL.

强化学习流匹配策略蒸馏探索机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。