用弱模型隐式奖励引导强模型生成,无需重训即可实现精准可控的图像生成。
Test-Time Weak-to-Strong Alignment: Transferring Implicit Rewards from Weak to Strong Flow Models

- 用弱模型对齐产生的隐式信号作为指导,跨规模迁移至强模型采样过程。
- 在不依赖奖励函数梯度的情况下,使大模型性能超越源对齐模型且保持高保真度。
- 一次训练可服务整个模型家族,适合需快速适配新任务的生成应用。
将文本到图像生成流模型与奖励对齐,使其遵循训练数据未提供的目标。对齐微调通过强化学习或偏好优化实现,但需为每个检查点重复训练,并固定于训练时的奖励与强度。测试时对齐则在采样过程中动态引导冻结模型,实现任务和样本级控制。现有方法仅通过奖励函数梯度或独立训练的价值函数获取每步信号。本文提出改变监督来源:使用一对弱模型而非奖励函数提供监督。一个对齐后的源模型与其基础模型共同构成源对齐对,将训练奖励以步骤级、KL锚定的隐式信号形式存储在采样器自身坐标中。我们探索该模型形式监督是否可跨规模迁移,结果表明可以:所提方法AlignGraft在采样中引入该对的速度差,对更大的冻结模型进行对齐。在共享噪声核下传输精确,测试时无需奖励或其梯度。方法无调度,仅需单一标量控制对齐强度,且可外推至超过源对齐对的强度。在图像与视频流模型(Stable Diffusion 3.5、FLUX、Wan)上,该方法在偏好、组合性与文本渲染奖励上均提升冻结大模型表现,甚至超越源对齐模型,同时以恒定小开销保持大模型保真度。大量实验表明,对弱模型的一次对齐即可产生可供全模型族在测试时复用的监督信号。
原文摘要 · Abstract (English)
Aligning a text-to-image generation flow model with a reward makes it follow objectives that the training data alone does not provide. Alignment fine-tuning delivers this by reinforcement learning (RL) or preference optimization, but it must be repeated for every checkpoint and returns a model fixed at the reward and strength it was trained with. Test-time alignment instead steers a frozen model during sampling, allowing task-specific and sample-specific guidance. Existing methods obtain this only by drawing the per-step signal from the reward function itself, through its gradient, or through a separately trained value function. We propose changing the supervision source: let a pair of weak models, not a reward function, supply the supervision. A source aligned model, kept together with its base as a source alignment pair, stores its training reward as an implicit, step-wise, KL-anchored signal expressed in the sampler's own coordinates. We explore whether this model-form supervision can cross scale, and show that it does: our method, AlignGraft, aligns a larger, frozen, never-tuned model by adding the pair's velocity difference during sampling. The transport is exact under a shared noising kernel and needs neither the reward nor its gradient at test time. The method has no schedules, only a single scalar that controls the alignment strength and can extrapolate it beyond that of the source alignment pair. Across image and video flow models (Stable Diffusion 3.5, FLUX, and Wan), the transfer lifts the frozen large model on preference, compositional, and text-rendering rewards, can exceed the source aligned model itself, and preserves the large model's fidelity at a small constant sampling overhead. Extensive experiments show that one alignment run on a weak model produces supervision that the whole model family can reuse at test time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。