让大模型自己选关键片段，提升长视频理解效率与准确率
Self-ReS: Self-Reflection in Large Vision-Language Models for Long Video Understanding
- 基于模型自身注意力图动态筛选关键视频片段
- 在相同显存下实现最高46%的推理加速，准确率显著提升
- 无需额外训练或模块，可直接接入现有大模型
大型视觉语言模型（LVLMs）在短视频任务中表现优异，但在长视频理解上存在困难。传统线性帧采样方式无法适应视频中关键事件的非线性分布，导致长视频中冗余信息过多，短视频中关键事件易被遗漏。为此，我们提出SelfReS——一种非线性时空自反思采样方法，根据用户提示动态选择关键视频片段。不同于以往方法，SelfReS利用LVLM固有的稀疏注意力图生成反射标记，实现感知相关性的标记选择，无需额外训练或外部模块。实验表明，SelfReS可无缝集成至强基线LVLM，提升长视频任务准确率，并在相同GPU内存预算下实现最高46%的推理速度提升。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) demonstrate remarkable performance in short-video tasks such as video question answering, but struggle in long-video understanding. The linear frame sampling strategy, conventionally used by LVLMs, fails to account for the non-linear distribution of key events in video data, often introducing redundant or irrelevant information in longer contexts while risking the omission of critical events in shorter ones. To address this, we propose SelfReS, a non-linear spatiotemporal self-reflective sampling method that dynamically selects key video fragments based on user prompts. Unlike prior approaches, SelfReS leverages the inherently sparse attention maps of LVLMs to define reflection tokens, enabling relevance-aware token selection without requiring additional training or external modules. Experiments demonstrate that SelfReS can be seamlessly integrated into strong base LVLMs, improving long-video task accuracy and achieving up to 46% faster inference speed within the same GPU memory budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。