评估视觉语言模型在无监督情况下的泛化能力,发现投影层对新概念仍有80%以上性能。
Prompt the Unseen: Evaluating Visual-Language Alignment Beyond Supervision
- 设计新基准,用分离标签集的提示格式测试投影层泛化能力。
- 在未见类别上仍保持79%-88%的性能,表明无需显式标注也能泛化。
- 揭示投影层像键值记忆,处理已见与未见概念方式相似,适合研究模型机制者。
视觉语言模型(VLMs)通过对齐训练结合视觉编码器与大语言模型(LLM),在多模态任务中表现优异。其核心组件——投影层将视觉特征映射到LLM嵌入空间——虽重要,但其对未见视觉概念的泛化能力尚未系统评估。为此,本文提出一个新基准:将细粒度标注的对象检测数据集转换为提示格式,并设计训练/测试集标签不重叠,实现对已见与未见概念的精确分离。实验结果表明,投影层在未见类别上的性能保持在已见类别的79%至88%之间,显示出非平凡的泛化能力,即使未对这些概念进行显式对齐监督。进一步通过机制可解释性分析发现,投影层中的前馈网络表现如键值记忆,对已见和未见令牌的处理方式相似。本研究引入了对齐泛化的新型评估框架,凸显了在有限对齐数据下高效训练VLM的潜力。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) combine a vision encoder and a large language model (LLM) through alignment training, showing strong performance on multimodal tasks. A central component in this architecture is the projection layer, which maps visual features into the LLM's embedding space. Despite its importance, its ability to generalize to unseen visual concepts has not been systematically evaluated. To address this, we propose a benchmark for evaluating projection-layer generalization. We adapt object detection datasets (rich in fine-grained annotations) into a prompting format and design train/test splits with disjoint label sets, enabling precise control over seen and unseen concept separation. Experimental results show that the projection layer retains about 79 to 88 percent of the performance on unseen classes compared to seen ones across various settings, suggesting a non-trivial level of generalization even without explicit alignment supervision on those concepts. We further analyze this behavior through a mechanistic interpretability lens. Our findings indicate that the feed-forward network in the projection layer functions like a key-value memory, processing seen and unseen tokens in similar ways. This study introduces a new evaluation framework for alignment generalization and highlights the potential for efficient VLM training with limited aligned data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。