研究视觉语言模型中图像嵌入投影导致的信息损失，发现其严重影响模型性能。
Lost in Embeddings: Information Loss in Vision-Language Models
- 通过分析图像嵌入在投影前后的最近邻关系，评估语义信息保留程度
- 投影后局部几何结构扭曲40%~60%，与检索性能下降直接相关
- 可定位高信息损失区域，帮助理解模型在视觉问答中的失败原因
视觉-语言模型（VLMs）通常通过预训练视觉编码器处理图像输入，再经连接模块投影到语言模型的嵌入空间。尽管该投影对模态融合至关重要，但其引发的信息损失及其对模型能力的影响尚未被充分研究。本文提出两种互补方法：首先，通过分析图像表示在投影前后最近邻关系的变化，评估语义信息的保留情况；其次，直接从投影后的表示中重建视觉嵌入，实现图像块级别的信息损失定位。实验表明，连接模块显著扭曲了视觉表示的局部几何结构，最近邻关系在投影后偏离40%~60%，且与检索性能下降密切相关。基于块级嵌入重建的结果，可解释模型在视觉基础问答任务中的行为，发现高信息损失区域能可靠预测模型出错的实例。
原文摘要 · Abstract (English)
Vision--language models (VLMs) often process visual inputs through a pretrained vision encoder, followed by a projection into the language model's embedding space via a connector component. While crucial for modality fusion, the potential information loss induced by this projection step and its direct impact on model capabilities remain understudied. We introduce two complementary approaches to examine and quantify this loss by analyzing the latent representation space. First, we evaluate semantic information preservation by analyzing changes in k-nearest neighbor relationships between image representations, before and after projection. Second, we directly measure information loss by reconstructing visual embeddings from the projected representation, localizing loss at an image patch level. Experiments reveal that connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40--60\% post-projection, correlating with degradation in retrieval performance. The patch-level embedding reconstruction provides interpretable insights for model behavior on visually grounded question-answering tasks, finding that areas of high information loss reliably predict instances where models struggle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。