让视觉模型持续关注关键物体,提升理解能力
Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Models
- 通过跨层视觉记忆平滑注意力分布,保持对关键物体的关注
- 在4个基准上实现最优综合表现,图像描述任务媲美顶尖方法
- 适合需要精准视觉理解的多模态应用,如智能助手、医疗影像分析
大型视觉语言模型(LVLMs)虽能准确定位图像中的关键物体,但对其注意力持续时间较短。基于持续关注有助于提升视觉能力的假设,我们提出跨层视觉平滑(CLVS)。其核心思想是引入一个视觉记忆,使注意力在各层间平滑分布:首层以位置无关的视觉注意力初始化记忆;后续层中,模型注意力同时参考前层记忆,并迭代更新记忆,从而维持对关键物体的稳定关注。鉴于视觉理解主要发生在模型早期与中间层,我们以不确定性作为视觉理解完成的指标,适时终止平滑过程。在三个LVLM上的四个基准测试验证了该方法的有效性与泛化能力。CLVS在多种视觉理解任务中达到当前最佳整体性能,在图像描述任务上与领先方法相当。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) can accurately locate key objects in images, yet their attention to these objects tends to be very brief. Motivated by the hypothesis that sustained focus on key objects can improve LVLMs' visual capabilities, we propose Cross-Layer Vision Smoothing (CLVS). The core idea of CLVS is to incorporate a vision memory that smooths the attention distribution across layers. Specifically, we initialize this vision memory with position-unbiased visual attention in the first layer. In subsequent layers, the model's visual attention jointly considers the vision memory from previous layers, while the memory is updated iteratively, thereby maintaining smooth attention on key objects. Given that visual understanding primarily occurs in the early and middle layers of the model, we use uncertainty as an indicator of completed visual understanding and terminate the smoothing process accordingly. Experiments on four benchmarks across three LVLMs confirm the effectiveness and generalizability of our method. CLVS achieves state-of-the-art overall performance across a variety of visual understanding tasks and attains comparable results to the leading approaches on image captioning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。