arXiv:2503.17142cs.CVcs.LG2025-03CVPR被引 8

发现视觉语言模型在图像表征中存在类人组合性,提升可解释性与鲁棒性。

Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models

  • 提出几何感知的组合分解框架(GDE),捕捉图像嵌入的非线性结构
  • 在组合分类和群体鲁棒性任务上优于传统线性方法,尤其在后者表现突出
  • 揭示视觉表征具备人类般的组合推理能力,适合研究模型可解释性者

视觉语言模型(VLMs)在文本与图像间建立共享特征空间,实现跨模态比较。已有研究显示,这些模型在语言表征中形成编码复合语义的规则结构,但图像表征是否也具备类似组合性尚不明确。本文针对视觉数据噪声大、稀疏性强的问题,提出几何可分解嵌入(GDE)框架,通过几何感知方式逼近图像嵌入中的组合结构。实验表明,预训练VLM的视觉嵌入确实呈现组合排列;在组合分类与群体鲁棒性任务中,GDE优于假设线性几何的基线方法。尤其在群体鲁棒性任务上,其性能超过专用解决方案。结果表明,VLM能在视觉域自动发展出类人的组合推理能力,使内部机制更可解释。代码已开源:https://github.com/BerasiDavide/vlm_image_compositionality。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) learn a shared feature space for text and images, enabling the comparison of inputs of different modalities. While prior works demonstrated that VLMs organize natural language representations into regular structures encoding composite meanings, it remains unclear if compositional patterns also emerge in the visual embedding space. In this work, we investigate compositionality in the image domain, where the analysis of compositional properties is challenged by noise and sparsity of visual data. We address these problems and propose a framework, called Geodesically Decomposable Embeddings (GDE), that approximates image representations with geometry-aware compositional structures in the latent space. We demonstrate that visual embeddings of pre-trained VLMs exhibit a compositional arrangement, and evaluate the effectiveness of this property in the tasks of compositional classification and group robustness. GDE achieves stronger performance in compositional classification compared to its counterpart method that assumes linear geometry of the latent space. Notably, it is particularly effective for group robustness, where we achieve higher results than task-specific solutions. Our results indicate that VLMs can automatically develop a human-like form of compositional reasoning in the visual domain, making their underlying processes more interpretable. Code is available at https://github.com/BerasiDavide/vlm_image_compositionality.

视觉语言模型组合性可解释性嵌入结构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。