用噪声对齐梯度，让生成模型更快更稳地提升奖励得分。
Gradient Preconditioning for Efficient and Reliable Reward-Guided Generation
- 通过投影梯度到白噪声可行集，实现噪声对齐的更新方向。
- 在FLUX上仅用30%时间达到与最优方法相当的美学评分。
- 适合追求高效可靠奖励驱动生成的研究者和开发者。
我们提出一种梯度预处理方法，使基于单步生成模型的奖励引导生成既高效又可靠。测试时的噪声优化能显著提升预训练生成模型的奖励引导效果，但易引发奖励滥用导致质量下降，且通常速度过慢难以实用。我们通过将奖励梯度投影到精心设计的白高斯噪声可行集（一个具有块状范数约束的紧凑谱集，紧密捕捉白高斯噪声的统计特性与空间无相关性）来实现预处理。该预处理将每次梯度更新重塑为噪声对齐方向，加速有效奖励上升并防止奖励滥用。投影为闭式解，复杂度匹配FFT的$O(N /log N)$，实际开销可忽略。在FLUX上使用四种奖励模型的实验表明，本方法仅需状态领先正则化方法30%的墙钟时间即可达到相当的美学评分。
原文摘要 · Abstract (English)
We propose a gradient preconditioning method that makes reward-guided generation with one-step generative models both efficient and reliable. Test-time noise optimization can unlock substantially better reward-guided generations from pretrained generative models, but it is prone to reward hacking that degrades quality and is often too slow for practical use. We precondition reward gradients by projecting them onto a carefully designed white Gaussian noise feasible set, a compact spectral set with blockwise norm constraints that tightly captures the statistics and spatial uncorrelatedness of white Gaussian noise. This preconditioning reshapes each gradient update into a noise-aligned direction, driving faster and more effective reward ascent while preventing reward hacking. The projection is closed-form and matches the $O(N \log N)$ complexity of FFT, adding negligible overhead in practice. In experiments on FLUX with four reward models, our approach reaches a comparable Aesthetic Score using only 30% of the wall-clock time required by the state-of-the-art regularization-based method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。