用随机一步采样加速生成模型对齐，大幅降低计算开销。
Meta Flow Maps enable scalable reward alignment
- 引入元流映射，实现任意中间状态的随机一步后验采样。
- 单粒子采样器在ImageNet上以极低算力超越1000次采样基线。
- 适用于推理时控制和无偏离线微调，适合高效奖励对齐场景。
控制生成模型计算成本高昂，因最优奖励对齐（如推理时调控或微调）需估计价值函数，而这通常依赖于条件后验分布 $p_{1|t}(x_1|x_t)$，即与中间状态 $x_t$ 一致的干净数据 $x_1$ 的分布，该需求常导致高成本轨迹模拟。为此，我们提出元流映射（Meta Flow Maps, MFMs），将一致性模型与流映射扩展至随机情形。MFMs 经训练可执行随机一步后验采样，从任意中间状态生成任意多独立同分布的干净数据 $x_1$ 样本。关键在于，这些样本提供可微重参数化，从而实现高效价值函数估计。我们利用此能力解决两类范式瓶颈：无需内层滚动即可实现推理时调控，且支持针对通用奖励的无偏、离线策略微调。实证表明，我们的单粒子调控-MFM采样器在ImageNet上，对多种奖励均显著优于基于最佳1000次采样的基线，且仅需极小算力。
原文摘要 · Abstract (English)
Controlling generative models is computationally expensive. This is because optimal alignment with a reward function--whether via inference-time steering or fine-tuning--requires estimating the value function. This task demands access to the conditional posterior $p_{1|t}(x_1|x_t)$, the distribution of clean data $x_1$ consistent with an intermediate state $x_t$, a requirement that typically compels methods to resort to costly trajectory simulations. To address this bottleneck, we introduce Meta Flow Maps (MFMs), a framework extending consistency models and flow maps into the stochastic regime. MFMs are trained to perform stochastic one-step posterior sampling, generating arbitrarily many i.i.d. draws of clean data $x_1$ from any intermediate state. Crucially, these samples provide a differentiable reparametrization that unlocks efficient value function estimation. We leverage this capability to solve bottlenecks in both paradigms: enabling inference-time steering without inner rollouts, and facilitating unbiased, off-policy fine-tuning to general rewards. Empirically, our single-particle steered-MFM sampler outperforms a Best-of-1000 baseline on ImageNet across multiple rewards at a fraction of the compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。