通过潜空间反射增强长视觉上下文中的细粒度感知能力
VisReflect: Latent Visual Reflection for Fine-Grained Perception in Long Visual Context

- 在潜空间生成连续视觉反射,引导注意力聚焦关键区域
- 图像与视频基准上分别提升4.1%和1.8%准确率
- 单次前向传播完成,推理速度比缩放方法快44%
大视觉语言模型在视觉-语言任务中表现优异,但在高分辨率图像和长视频上下文中进行细粒度感知仍具挑战。随着视觉标记数量增加,视觉注意力下沉现象愈发严重,导致无关标记吸收过多注意力。现有方法通过预测边界框或时间区间并重新编码裁剪区域来缓解此问题,但依赖离散标记空间中的不可靠定位,且需额外前向传播,计算开销大。本文提出VisReflect,一种简单高效的框架,通过潜空间视觉反射提升长视觉上下文中的细粒度感知。模型不将中间预测解码为离散标记,而是在潜空间生成连续视觉反射,表征与问题相关的视觉特征。这些反射可选择性强调显著区域或帧,引导注意力集中于相关视觉标记,仅需一次前向传播。我们在多个高分辨率图像基准(BLINK、V*、HRBench-4K/8K)及视频理解基准(MVBench、VideoMME、MLVU)上进行评估,结果表明该方法持续优于强基线,在图像基准上提升4.1%,视频基准上提升1.8%。相比基于缩放的方法,性能相当但视频理解推理时间减少约44%。
原文摘要 · Abstract (English)
Large Vision Language Models (LVLMs) have achieved remarkable success on vision-language tasks, yet fine-grained perception over high-resolution images and long-context videos remains challenging. As the number of visual tokens increases, the visual attention sink phenomenon becomes increasingly severe, causing irrelevant tokens to absorb a disproportionate amount of attention mass. Recent approaches attempt to mitigate this issue by explicitly predicting bounding boxes or temporal spans and re-encoding the cropped visual regions. Such methods depend on unreliable numeric localization in the discrete token space and incur significant computational overhead due to additional forward passes. In this work, we propose **VisReflect**, a simple yet effective framework that improves fine-grained perception in long visual contexts through latent visual reflection. Instead of decoding intermediate predictions into discrete tokens, the model generates continuous visual reflection that represents question-relevant visual features in the latent space. These reflections selectively emphasize salient regions or frames, guiding attention towards relevant visual tokens within a single forward pass. We conduct comprehensive evaluations on challenging high-resolution image benchmarks, including BLINK, V*, and HRBench-4K/8K, as well as video understanding benchmarks such as MVBench, VideoMME, and MLVU. Our method consistently improves over strong baselines, achieving gains of 4.1% on image benchmarks and 1.8% on video benchmarks. Compared with zooming-based methods, our model achieves comparable performance while reducing inference time by roughly 44% on video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。