通过调整噪声方向实现奖励对齐，不改生成路径却大幅提升效率。
NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignment
- 用噪声项注入奖励梯度，保持原反向过程不变
- 25次采样即达基线500次效果，计算量降20倍
- 适合需要高效高质生成的奖励对齐任务
我们提出噪声倾斜反向核（NTRK），一种在推理时进行奖励引导的扩散采样方法。它通过噪声项注入奖励梯度，保持预训练反向核不变，且每步仅需一次采样。现有方法存在权衡：基于梯度的引导会偏移反向均值，导致中间状态偏离训练分布，降低质量；基于搜索的方法虽保质但无梯度信号。NTRK通过固定反向均值、倾斜噪声项来引导高奖励方向，其核心是白化算子，将奖励梯度转化为与噪声兼容的扰动，同时保留引导信号。在多种奖励对齐任务中，NTRK超越近期最优基线，且不损失样本质量。令人惊喜的是，在审美生成任务中，仅用25次NFE（非自回归步数）即超越基线在500次NFE下的奖励表现，计算量降低20倍。
原文摘要 · Abstract (English)
We introduce the Noise-Tilted Reverse Kernel (NTRK), a reward-guided diffusion sampler that injects reward gradients through the noise term, leaving the pretrained reverse kernel unchanged and requiring only a single sample per step. Reward-guided sampling at inference time has greatly expanded the versatility of pretrained diffusion models. Yet existing methods face a trade-off. Gradient-based guidance shifts the reverse mean, steering generation but pushing intermediate states outside the region that the model was trained on and degrading quality. Search-based methods preserve quality but gain no gradient signal. No prior method achieves both. NTRK resolves this by keeping the reverse mean fixed and biasing the noise term toward high reward. This is enabled by a whitening operator, the central mechanism behind NTRK, which converts reward gradients into noise-compatible perturbations without losing their guiding signal. Across various reward alignment tasks, NTRK outperforms recent state-of-the-art baselines without losing sample quality. Remarkably, on aesthetic generation, NTRK surpasses the reward of the best baseline at 500 NFEs using only 25 NFEs, a 20 times reduction in compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。