用扩散模型生成的表示增强零样本组合识别,提升泛化能力。
DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations

- 引入扩散模型中间表示作为辅助监督信号
- 在三个基准上均超越现有CLIP基线方法
- 无需额外计算成本,适合视觉语言模型研究者
组合零样本学习(CZSL)旨在通过已知属性-对象组合中习得的原始概念知识,识别未见过的组合。尽管近期工作借助大型视觉语言模型取得显著进展,但其主要依赖判别式表示,可能未能显式保留原始概念与组合间的结构关系。受扩散分类器成功启发,我们探究中间扩散表示是否可为CZSL提供互补线索。为此,提出DIFFCZSL:一种将预训练扩散模型的生成先验注入基于CLIP的CZSL流程的框架。通过提取中间扩散表示并投影至CLIP嵌入空间,在图像与文本模态上提供辅助监督。训练过程中,通过对比对齐CLIP嵌入与扩散特征,促使嵌入几何趋向更丰富的组合感知语义,且推理时无额外开销。在三个公开CZSL基准上的大量实验表明,该方法在封闭世界与开放世界设置下均持续优于强基线。结果凸显了生成式扩散表示与判别式视觉语言模型在组合泛化中的互补优势。
原文摘要 · Abstract (English)
Compositional Zero-Shot Learning (CZSL) aims to recognize unseen attribute-object compositions by leveraging knowledge of primitive concepts learned from seen compositions. Although recent works achieve impressive performance in CZSL by leveraging large vision-language models, they primarily rely on discriminative representations that may not explicitly preserve the structured relationships between primitive concepts and their compositions. Motivated by the recent success of diffusion-based classifiers and their competitive performance relative to discriminative models, we investigate whether intermediate diffusion representations can provide complementary cues for CZSL. To this end, we propose DIFFCZSL, a diffusion-augmented framework that injects generative priors from pre-trained diffusion models into CLIP-based CZSL pipelines. We extract intermediate diffusion representations and project them into the CLIP embedding space to provide auxiliary supervision on both image and text modalities. Through contrastive alignment between CLIP embeddings and diffusion features during training, our method encourages the embedding geometry toward richer composition-aware semantics, while introducing no additional cost at inference time. Extensive experiments on three public CZSL benchmarks demonstrate consistent improvements over strong CLIP-based baselines under both closed-world and open-world settings. Our results highlight the complementary strengths of generative diffusion representations and discriminative vision-language models for compositional generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。