arXiv:2508.20783cs.CVcs.AI2025-08中稿 · IWCS 2025被引 4

对比扩散模型与视觉语言模型的组合泛化能力,发现两者在关系推理上仍有局限。

Evaluating Compositional Generalisation in VLMs and Diffusion Models

  • 用扩散分类器替代传统判别模型,提升对象属性绑定能力
  • 三类模型在关系泛化任务中表现均差,准确率不足40%
  • 适合关注模型语义理解局限性的研究者阅读

自然语言的语义核心在于通过已知成分组合生成新意义。尽管视觉语言模型(VLMs)近年进展显著,但证据表明其难以实现此类组合推理。例如,面对红方块与蓝圆柱的图像,CLIP等模型常误标为红圆柱或蓝方块,反映出其将图像视为“词袋”而未能捕捉组合语义。扩散模型因其生成能力强受到关注,基于扩散的零样本分类器在某些组合任务中已可与CLIP比肩。本文评估扩散分类器、CLIP与ViLT在零样本学习(ZSL)和广义零样本学习(GZSL)下绑定物体属性与关系的能力。结果显示,扩散分类器与ViLT在概念绑定任务中表现良好,但所有模型在关系型GZSL任务中均表现不佳,准确率低于40%,揭示了当前VLMs在关系推理上的普遍挑战。对CLIP嵌入的分析表明,问题可能源于左/右等关系概念的表示过于相似。代码与数据集见:https://github.com/otmive/diffusion_classifier_clip

原文摘要 · Abstract (English)

A fundamental aspect of the semantics of natural language is that novel meanings can be formed from the composition of previously known parts. Vision-language models (VLMs) have made significant progress in recent years, however, there is evidence that they are unable to perform this kind of composition. For example, given an image of a red cube and a blue cylinder, a VLM such as CLIP is likely to incorrectly label the image as a red cylinder or a blue cube, indicating it represents the image as a `bag-of-words' and fails to capture compositional semantics. Diffusion models have recently gained significant attention for their impressive generative abilities, and zero-shot classifiers based on diffusion models have been shown to perform competitively with CLIP in certain compositional tasks. In this work we explore whether the generative Diffusion Classifier has improved compositional generalisation abilities compared to discriminative models. We assess three models -- Diffusion Classifier, CLIP, and ViLT -- on their ability to bind objects with attributes and relations in both zero-shot learning (ZSL) and generalised zero-shot learning (GZSL) settings. Our results show that the Diffusion Classifier and ViLT perform well at concept binding tasks, but that all models struggle significantly with the relational GZSL task, underscoring the broader challenges VLMs face with relational reasoning. Analysis of CLIP embeddings suggests that the difficulty may stem from overly similar representations of relational concepts such as left and right. Code and dataset are available at: https://github.com/otmive/diffusion_classifier_clip

视觉语言模型组合泛化关系推理扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。