提出轻量级注意力可视化方法，揭示视觉语言模型如何关注图表区域生成答案。
Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMs
- 通过聚合多头多层注意力，将文本生成与图像区域直接对应。
- 用删除评估验证注意力图与模型行为的因果一致性。
- 无需梯度计算，快速生成可解释的注意力热图，适合分析模型推理过程。
理解视觉语言模型（VLMs）如何解析数据可视化仍是开放问题，尤其在需要可靠推理的分析任务中愈发重要。本文提出一种面向基于Transformer的文本生成图像解释任务的轻量级诊断显著性图方法。该方法聚合语言模型在所有注意力头和层上对视觉标记的注意力，并将其映射回视觉编码器的图像块网格，实现生成答案中的每个词与图像区域的直接对应。由此产生的显著性图无需梯度计算，可快速揭示VLM在生成答案时对视觉元素的关注分布，从而检验其注意力是否聚焦于语义相关区域。我们使用删除指标评估方法，验证了显著性图与模型行为之间的因果忠实性。
原文摘要 · Abstract (English)
Understanding how vision-language models (VLMs) interpret data visualizations remains an open problem, and is increasingly important as these models are used for analytical tasks where reliable reasoning is essential. We introduce a lightweight, diagnostic saliency map method tailored for text generation over images using transformer models, the current state-of-the-art models in visualization interpretation. Our approach aggregates the language model's attention over the visual tokens across all heads and layers, then maps this attention back onto the vision encoder's patch grid to localise it over the image, producing a direct correspondence between each generated answer token and the image regions it attended to. This yields fast, gradient-free saliency maps that expose how VLMs allocate focus across visual elements during answer generation, enabling inspection of whether model attention aligns with semantically relevant components. We evaluate our approach using a deletion metric which validates the causal faithfulness of our saliency maps to the model's behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。