通过视觉子空间结构强化跨帧视觉线索，提升多模态模型的空间推理能力
VideoAnchor: Reinforcing Subspace-Structured Visual Cues for Coherent Visual-Spatial Reasoning
- 利用稀疏子空间聚类的自表达性，设计可插拔模块增强跨帧视觉注意力
- 在VSI-Bench和Video-MME上分别取得3.2%和4.6%的性能提升
- 适用于需精准视觉空间对齐的视频理解任务，如视觉问答与定位
多模态大语言模型在视觉-语言对齐方面已取得显著进展，但在视觉空间推理方面仍受限。我们发现该局限源于注意力机制：视觉标记被语言标记压制，导致模型难以在不同帧间持续识别同一视觉线索。为此，我们首次将稀疏子空间聚类中的自表达性特性与Transformer注意力机制相联系。基于此，提出VideoAnchor——一种无需微调即可强化跨帧视觉线索的即插即用模块，有效锚定注意力于共享视觉结构。在多个基准和骨干模型上的实验表明，性能稳定提升：例如，在VSI-Bench和Video-MME（空间相关任务）上，使用InternVL2-8B和Qwen2.5VL-72B分别获得3.2%和4.6%的增益。定性分析显示，子空间划分更连贯，视觉定位更准确。代码将公开于https://github.com/feufhd/VideoAnchor。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved impressive progress in vision-language alignment, yet they remain limited in visual-spatial reasoning. We first identify that this limitation arises from the attention mechanism: visual tokens are overshadowed by language tokens, preventing the model from consistently recognizing the same visual cues across frames. To address this challenge, we draw a novel connection between the self-expressiveness property in sparse subspace clustering and the attention mechanism in Transformers. Building on this insight, we propose VideoAnchor, a plug-and-play module that leverages subspace affinities to reinforce visual cues across frames without retraining, effectively anchoring attention to shared visual structures. Extensive experiments across benchmarks and backbone models show consistent performance gains -- $e.g.$, 3.2% and 4.6% improvements on VSI-Bench and Video-MME (spatial-related tasks) with InternVL2-8B and Qwen2.5VL-72B -- while qualitative analyses demonstrate more coherent subspace partitions and stronger visual grounding. Our codes will be made public available at https://github.com/feufhd/VideoAnchor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。