在视觉编码器中加入堆叠时序注意力，提升视频大模型对动作序列的理解能力。
Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders
- 将堆叠时序注意力嵌入视觉编码器，捕捉帧间动态关系
- 在多个基准上提升5.5%的视频问答准确率，尤其改善动作识别
- 适合需要精细时序理解的视频分析任务，如复杂行为推理
尽管多模态大语言模型取得了显著进展，但理解视频中的复杂时序动态仍是重大挑战。实验表明，当前视频大语言模型在需要详细理解动作序列和时间演进的任务上存在明显局限。本文提出一种新架构，在视觉编码器中引入堆叠时序注意力模块，使模型能在将视觉标记传递给语言模型前，更有效地捕捉动作发展过程及帧间关联。结果表明，该方法显著提升时序推理能力，在视频问答任务中表现优于现有模型，特别是在动作识别方面。在VITATECS、MVBench和Video-MME等多个基准上，性能最高提升达+5.5%。通过增强视觉编码器的时序结构，有效填补了视频大语言模型在视频理解方面的关键短板。项目页面与代码已公开：https://alirasekh.github.io/STAVEQ2/
原文摘要 · Abstract (English)
Despite significant advances in Multimodal Large Language Models (MLLMs), understanding complex temporal dynamics in videos remains a major challenge. Our experiments show that current Video Large Language Model (Video-LLM) architectures have critical limitations in temporal understanding, struggling with tasks that require detailed comprehension of action sequences and temporal progression. In this work, we propose a Video-LLM architecture that introduces stacked temporal attention modules directly within the vision encoder. This design incorporates a temporal attention in vision encoder, enabling the model to better capture the progression of actions and the relationships between frames before passing visual tokens to the LLM. Our results show that this approach significantly improves temporal reasoning and outperforms existing models in video question answering tasks, specifically in action recognition. We improve on benchmarks including VITATECS, MVBench, and Video-MME by up to +5.5%. By enhancing the vision encoder with temporal structure, we address a critical gap in video understanding for Video-LLMs. Project page and code are available at: https://alirasekh.github.io/STAVEQ2/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。