让视觉语言模型的图像片段解释更精准
Logit Lens Supervision for Patch-Level Explanations in Vision-Language Models
- 用词向量投影法追踪视觉令牌与图像区域的关联
- 提升对象定位准确率，减少幻觉错误
- 无需修改结构，适合快速部署到现有模型
现代自回归视觉语言模型虽能生成流畅回答，但其视觉令牌表示与原始图像区域的关联变弱，限制了像素级可解释性。本文通过Logit Lens映射，将每个视觉令牌嵌入通过大语言模型词表头投影，衡量图像区域与查询概念（如'猫'）的关联强度。提出轻量级辅助损失Logit Lens Loss（LLL），直接对齐与物体相关的视觉令牌与描述其区域的词汇概念，以保持局部视觉语义。LLL无需架构修改、掩码解码器或大规模重训练。在LLaVA-v1.5-7B和Qwen2.5-VL-7B上评估显示，使用LLL后，对象置信度图更清晰，定位和幻觉指标改善，零样本指代任务性能提升，且问答性能保持不变。
原文摘要 · Abstract (English)
Modern autoregressive Vision-Language Models (VLMs) can generate fluent answers while their visual-token representations become weakly tied to the image regions from which they originate. This limits patch-level explainability: a visual token should remain interpretable as the image patch it represents. We study this issue through Logit Lens maps, obtained by projecting each visual-token embedding through the LLM vocabulary head to measure how strongly each image patch is associated with a queried textual concept, such as "cat". We introduce Logit Lens Loss (LLL), a lightweight auxiliary objective that preserves localized visual semantics by directly aligning object-related visual tokens with the vocabulary concepts describing their image regions. LLL requires no architectural modification, mask decoder, or large-scale retraining. We evaluate LLL through both explanation quality and downstream performance. Across LLaVA-v1.5-7B and Qwen2.5-VL-7B, LLL yields sharper object confidence maps, improves grounding and hallucination metrics, transfers to zero-shot pointing, and preserves VQA performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。