arXiv:2603.26798cs.LGcs.AI2026-03中稿 · IJCAI

分析视觉语言模型嵌入空间的语义层级,发现图文编码器差异并提升语义对齐。

Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings

  • 通过聚类与词典匹配构建类别层次树,解释模型隐含语义结构。
  • 量化人类知识一致性,发现图像编码器更细粒度,文本编码器更贴近人类分类。
  • 提出轻量级对齐方法,用UMAP生成目标层级,适合改进模型可解释性。

视觉语言模型(如CLIP)在共享的图像-文本嵌入空间中实现了强大的检索与零样本分类能力,但该空间的语义组织方式极少被深入研究。本文提出一个后处理框架,用于解释、验证和对齐特定子类集合上由视觉语言模型诱导的语义层级。首先,通过类中心的凝聚聚类提取二叉层级结构,并利用词典匹配从概念库中命名内部节点。其次,通过高效的树级与边级一致性度量,将提取的树与人类本体进行比较以量化合理性;同时通过可解释的分层遍历推理与不确定性感知提前停止(UAES)评估实用性。第三,提出一种基于本体引导的后处理对齐方法,使用UMAP从理想层级生成目标邻域,学习轻量级嵌入空间变换。在13个预训练视觉语言模型和4个图像数据集上的实验表明:图像编码器更具区分性,而文本编码器的层级更符合人类分类体系。整体结果揭示了零样本准确率与本体合理性之间的持续权衡,并为改善共享嵌入空间中的语义对齐提供了可行路径。

原文摘要 · Abstract (English)

Vision-language model (VLM) encoders such as CLIP enable strong retrieval and zero-shot classification in a shared image-text embedding space, yet the semantic organization of this space is rarely inspected. We present a post-hoc framework to explain, verify, and align the semantic hierarchies induced by a VLM over a given set of child classes. First, we extract a binary hierarchy by agglomerative clustering of class centroids and name internal nodes by dictionary-based matching to a concept bank. Second, we quantify plausibility by comparing the extracted tree against human ontologies using efficient tree- and edge-level consistency measures, and we evaluate utility via explainable hierarchical tree-traversal inference with uncertainty-aware early stopping (UAES). Third, we propose an ontology-guided post-hoc alignment method that learns a lightweight embedding-space transformation, using UMAP to generate target neighborhoods from a desired hierarchy. Across 13 pretrained VLMs and 4 image datasets, our method finds systematic modality differences: image encoders are more discriminative, while text encoders induce hierarchies that better match human taxonomies. Overall, the results reveal a persistent trade-off between zero-shot accuracy and ontological plausibility and suggest practical routes to improve semantic alignment in shared embedding spaces.

视觉语言模型语义层级可解释性嵌入对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。