通过优化采样路径提升扩散模型对齐效果，训练快、推理无额外开销。
Diffusion-Sharpening: Fine-tuning Diffusion Models with Denoising Trajectory Sharpening
- 基于路径积分框架选择最优采样轨迹，利用奖励反馈进行优化。
- 训练收敛更快，推理无需增加采样步数（NFE），效率领先。
- 适合需要高效微调且注重生成质量的场景，如文本到图像生成。
我们提出 Diffusion-Sharpening，一种通过优化采样轨迹来增强下游对齐的微调方法。现有基于强化学习的微调方法仅关注单个训练时间步，忽略轨迹级对齐；而近期的采样轨迹优化方法则带来显著的推理 NFE 增加。Diffusion-Sharpening 通过路径积分框架，在训练中选择最优轨迹，结合奖励反馈，并分摊推理成本。实验表明，该方法在训练效率上更快收敛，推理效率最优，无需额外 NFE。在文本对齐、组合能力及人类偏好等多维度指标上，均优于 Diffusion-DPO 等 RL 微调方法和 Inference Scaling 等轨迹优化方法，为未来扩散模型微调提供可扩展、高效的解决方案。
原文摘要 · Abstract (English)
We propose Diffusion-Sharpening, a fine-tuning approach that enhances downstream alignment by optimizing sampling trajectories. Existing RL-based fine-tuning methods focus on single training timesteps and neglect trajectory-level alignment, while recent sampling trajectory optimization methods incur significant inference NFE costs. Diffusion-Sharpening overcomes this by using a path integral framework to select optimal trajectories during training, leveraging reward feedback, and amortizing inference costs. Our method demonstrates superior training efficiency with faster convergence, and best inference efficiency without requiring additional NFEs. Extensive experiments show that Diffusion-Sharpening outperforms RL-based fine-tuning methods (e.g., Diffusion-DPO) and sampling trajectory optimization methods (e.g., Inference Scaling) across diverse metrics including text alignment, compositional capabilities, and human preferences, offering a scalable and efficient solution for future diffusion model fine-tuning. Code: https://github.com/Gen-Verse/Diffusion-Sharpening
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。