arXiv:2605.23522cs.LGcs.AI2026-05

提出Precise采样器,让流匹配模型强化学习训练更快更稳。

Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models

论文配图:Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models
图 1 · 摘自论文原文
  • 设计SDE采样器平衡探索与稳定性,优化强化学习策略
  • 实验显示训练速度提升13.1%~53.2%,对齐分数达领先水平
  • 适合追求高效稳定训练的生成模型研究者

强化学习(RL)已成为提升扩散模型与流匹配生成器提示对齐性和感知质量的有效方法。将在线强化学习应用于流匹配的关键步骤是将确定性采样轨迹转化为随机策略,通常通过用随机微分方程(SDE)替代反向时间常微分方程(ODE)实现。该随机采样器作为策略的一部分,控制探索行为和去噪动态,其设计显著影响奖励优化性能。本文将采样器设计拆分为两个相互依赖的组件:选择合适的随机探索量,以及在强化学习中常用的小步数下忠实离散化所生成的SDE。针对第一部分,分析去噪过程中探索与稳定性的内在矛盾,推导出一个平衡两者的SDE调度策略。针对离散化挑战,通过简化示例表明现有采样器可能偏离流匹配过程,或引入过多离散化噪声,或依赖无法保证收敛至数据分布的启发式规则。为此,本文提出Precise新采样器,在保持有效探索的同时确保稳定性。关键在于采用新近似方法冻结干净潜在表示后验均值,解决了标准采样器的过度噪声问题。大量实验证明,该方法显著加速并稳定了强化学习下的奖励优化,在多项指标(如PickScore、HPSv2.1)上达到最先进水平,且所需墙钟时间比先前采样器减少13.1%至53.2%即可达成最佳域内性能。

原文摘要 · Abstract (English)

Reinforcement learning (RL) has become an effective way to improve prompt alignment and perceptual quality in diffusion and flow-matching generators. A critical step for applying online RL to flow matching is turning the deterministic sampling trajectory into a stochastic policy, typically by replacing the reverse-time Ordinary Differential Equation (ODE) with a Stochastic Differential Equation (SDE). The stochastic sampler, controlling the exploration behavior and denoising dynamics, is thus part of the policy, and its design can significantly affect the reward optimization performance. We break down the sampler design into two interdependent components: choosing the right amount of stochastic exploration, and discretizing the resulting SDE faithfully at the small step counts used in RL. To address the first component, we analyze the inherent tension between exploration and stability in denoising and derive an SDE schedule that balances the two. Turning to the discretization challenge, we use a toy example to show that existing samplers can deviate from the flow-matching process, either by introducing excessive discretization noise or by relying on heuristic rules that do not guarantee convergence to the data distribution. To address these issues, we propose Precise, a new stochastic sampler that balances effective exploration with stability. Crucially, Precise keeps the denoising trajectory SDE-consistent through a novel approximation that freezes the clean-latent posterior mean, resolving the excess noise issue in standard samplers. Extensive experiments demonstrate that this formulation leads to significantly faster and more stable reward optimization via reinforcement learning, achieving state-of-the-art alignment scores (e.g., PickScore, HPSv2.1) while requiring 13.1-53.2% less wall-clock training time to match the best in-domain performance of prior samplers.

强化学习流匹配采样器生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。