发现视觉语言模型描述图像时的注视机制，可精准控制其关注区域。
Gaze Heads: How VLMs Look at What They Describe
- 通过相关性分析定位模型中专注图像区域的特定注意力头。
- 仅干预前100个注视头，即可83.1%准确引导模型描述指定漫画面板。
- 无需重训练，可实时切换关注区域，适用于多种模型与图像类型。
视觉语言模型如何内部完成图像描述任务尚不明确。我们发现，模型发展出一种特定机制：语言模型主干中少数几个注意力头（称为注视头），其注意力会追踪当前描述的图像区域。通过少量前向传播计算相关性得分，并以漫画条为受控测试场景（叙事顺序在空间上排列），成功识别出这些注视头。这些注视头不仅跟踪正在描述的图像标记，还将注意力强制转向某区域，可使模型转而描述该区域。对前100个注视头进行单次注意力掩码干预（少于所有头的9%），即可在83.1%准确率下将模型输出引导至任意选定的漫画面板；相同操作作用于随机头则无效，全头干预则破坏生成。该控制机制还可实现连续调控：中途切换注视目标，模型在数个词内完成当前面板描述并转至新目标。此机制在自然图像（COCO）中同样有效。该现象在2B至32B参数量级的模型及多种架构中重复出现，但部分冻结编码器类模型未见类似头集。更广泛地，这表明基于机制分析识别出的精准干预点，可在推理阶段作为无需重训练的行为控制杠杆。代码、交互演示及数据集已公开于 https://gaze.baulab.info/
原文摘要 · Abstract (English)
How a vision-language model internally solves the task of describing an image is far from obvious. We find that the model develops a specific mechanism for this: a small set of attention heads in its language-model backbone, which we call gaze heads, whose attention tracks the image region the model is currently describing. We find them with a simple correlation score from a few forward passes, using comic strips as a controlled testbed where narrative order is laid out spatially. These gaze heads do not just track the image tokens being described: redirecting their attention to a chosen region forces the VLM to describe that region instead. A single attention-mask intervention on the top-100 gaze heads, fewer than 9% of all heads, steers the model's answer to any chosen comic panel at 83.1% accuracy, while the same intervention on random heads fails to redirect the answer, and intervening on all heads destroys generation. The same lever also extends to continuous control: switching the gaze target mid-generation makes the model wrap up its current panel description and move to the new one within a few tokens. Beyond comics, the same intervention redirects answers to chosen regions in natural COCO images. The mechanism further recurs across model sizes from 2B to 32B parameters and across other VLM architectures, although some frozen-encoder families show no comparable head set. More broadly, this shows that targeted edits identified through mechanistic analysis can serve as practical inference-time levers for steering multimodal model behavior, without any retraining. Our code, interactive demo, and datasets are available at https://gaze.baulab.info/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。