让大模型像人一样聚焦重点区域，提升视觉理解效率。
Learning to See What You Need: Gaze Attention for Multimodal Large Language Models
- 用轻量级注视区分组视觉特征，动态选择关注区域
- 在多个基准上表现优于或持平密集注意力，减少90%计算量
- 适合需要高效推理的多模态应用，如视频理解
人类描述视觉场景时不会均匀处理整个图像，而是有选择地聚焦于与描述相关的区域。相比之下，当前多模态大语言模型（MLLMs）在每一步生成中都会关注所有视觉标记，导致注意力分散并增加不必要的计算开销。本文提出Gaze Attention机制，使MLLMs在生成过程中能有选择性地关注任务相关的视觉区域。具体而言，将存储为键值缓存的视觉嵌入按空间分组为紧凑的注视区域，每个区域由一个轻量级描述符表示。在每一步解码时，模型动态选择最相关区域并限制注意力范围，从而减少冗余计算并增强关注焦点。为缓解局部注意力带来的全局上下文丢失问题，进一步引入可学习的上下文标记，附加到每张图像或帧上，帮助模型保持整体视觉感知。在图像和视频理解基准上的大量实验表明，Gaze Attention在性能上达到或超过密集注意力基线，同时在注意力计算中使用的视觉键值条目最多减少90%。
原文摘要 · Abstract (English)
When humans describe a visual scene, they do not process the entire image uniformly; instead, they selectively fixate on regions relevant to their intended description. In contrast, current multimodal large language models (MLLMs) attend to all visual tokens at each generation step, leading to diluted focus and unnecessary computational overhead. In this work, we introduce Gaze Attention, a novel mechanism that enables MLLMs to selectively attend to task-relevant visual regions during generation. Specifically, we spatially group visual embeddings-stored as key-value caches-into compact gaze regions, each represented by a lightweight descriptor. At each decoding step, the model dynamically selects the most relevant regions and restricts attention to them, reducing redundant computation while enhancing focus. To mitigate the loss of global context caused by localized attention, we further propose learnable context tokens appended to each image or frame, allowing the model to maintain holistic visual awareness. Extensive experiments on image and video understanding benchmarks demonstrate that Gaze Attention matches or surpasses dense-attention baselines, while using up to 90% fewer visual KV entries in the attention computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。