arXiv:2603.04803cs.CVcs.AI2026-03被引 2

用对比信号引导扩散重建,平衡模型判别力与细节感知能力。

Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation

  • 将对比学习信号注入扩散重建过程,优化视觉表示
  • 在多个基准上提升图像分类与多模态任务性能
  • 适合希望改进CLIP视觉编码器表示能力的研究者

CLIP的视觉编码器存在理解能力不足的问题,主要体现在类别区分能力(D-Ability)和细粒度感知能力(P-Ability)的局限。现有方法利用扩散模型基于CLIP视觉标记进行图像重建,但可能损害其判别力。为此,本文提出扩散对比重建(DCR),通过从重建图像中提取对比信号注入扩散过程,而非原始输入。理论分析表明,该损失可联合优化D-Ability与P-Ability。实验在多个基准及多模态大模型上验证了有效性,显著提升了下游任务表现。代码已开源。

原文摘要 · Abstract (English)

The limited understanding capacity of the visual encoder in Contrastive Language-Image Pre-training (CLIP) has become a key bottleneck for downstream performance. This capacity includes both Discriminative Ability (D-Ability), which reflects class separability, and Detail Perceptual Ability (P-Ability), which focuses on fine-grained visual cues. Recent solutions use diffusion models to enhance representations by conditioning image reconstruction on CLIP visual tokens. We argue that such paradigms may compromise D-Ability and therefore fail to effectively address CLIP's representation limitations. To address this, we integrate contrastive signals into diffusion-based reconstruction to pursue more comprehensive visual representations. We begin with a straightforward design that augments the diffusion process with contrastive learning on input images. However, empirical results show that the naive combination suffers from gradient conflict and yields suboptimal performance. To balance the optimization, we introduce the Diffusion Contrastive Reconstruction (DCR), which unifies the learning objective. The key idea is to inject contrastive signals derived from each reconstructed image, rather than from the original input, into the diffusion process. Our theoretical analysis shows that the DCR loss can jointly optimize D-Ability and P-Ability. Extensive experiments across various benchmarks and multi-modal large language models validate the effectiveness of our method. The code is available at https://github.com/boyuh/DCR.

扩散模型对比学习视觉表征CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。