让扩散模型更准地做视觉感知,不改结构也能提升性能。
Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception
- 针对不同去噪阶段设计差异化学习目标,提升感知精度。
- 发现后期去噪会降低准确率,用定制增强缓解分布偏移问题。
- 利用生成过程可交互特性,支持多轮修正提示,适合实际应用。
随着图像生成的成功,生成式扩散模型正被越来越多用于判别性任务,因其像素生成提供统一的感知接口。然而,直接将生成去噪过程用于判别目标时存在关键差距:生成模型容忍中间采样误差,只要最终分布合理即可;而判别任务要求全程高精度,尤其在引用图像分割等多模态挑战任务中表现明显。为此,我们分析并增强生成去噪过程与感知任务之间的对齐,聚焦去噪过程中感知质量的演变。发现:(1)早期去噪步骤对感知质量贡献更大,提出反映不同时步贡献的定制学习目标;(2)后期去噪步骤出现意外感知退化,归因于训练-去噪分布偏移,通过扩散定制数据增强解决;(3)生成过程天然支持交互,可作为可控用户界面,适应多轮修正提示。这些洞察在不改变模型结构的前提下显著提升基于扩散的感知模型性能,在深度估计、引用图像分割和通用感知任务上达到当前最优水平。代码已公开于 https://github.com/ziqipang/ADDP。
原文摘要 · Abstract (English)
With the success of image generation, generative diffusion models are increasingly adopted for discriminative tasks, as pixel generation provides a unified perception interface. However, directly repurposing the generative denoising process for discriminative objectives reveals critical gaps rarely addressed previously. Generative models tolerate intermediate sampling errors if the final distribution remains plausible, but discriminative tasks require rigorous accuracy throughout, as evidenced in challenging multi-modal tasks like referring image segmentation. Motivated by this gap, we analyze and enhance alignment between generative diffusion processes and perception tasks, focusing on how perception quality evolves during denoising. We find: (1) earlier denoising steps contribute disproportionately to perception quality, prompting us to propose tailored learning objectives reflecting varying timestep contributions; (2) later denoising steps show unexpected perception degradation, highlighting sensitivity to training-denoising distribution shifts, addressed by our diffusion-tailored data augmentation; and (3) generative processes uniquely enable interactivity, serving as controllable user interfaces adaptable to correctional prompts in multi-round interactions. Our insights significantly improve diffusion-based perception models without architectural changes, achieving state-of-the-art performance on depth estimation, referring image segmentation, and generalist perception tasks. Code available at https://github.com/ziqipang/ADDP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。