提出时间门控模块，提升视频大模型对时序信息的理解能力。
Enhancing Temporal Modeling of Video LLMs via Time Gating
- 设计时间门控模块，动态调节空间、时序注意力与MLP的输出。
- 在MVBench、TempCompass等基准上显著超越现有视频大模型。
- 适合需要精准时序理解的视频问答与分析任务。
视频大语言模型（Video LLMs）在视频-语言任务（如视频问答）中表现优异，但多数模型忽视视频数据中的时序信息，导致难以进行时序感知的视频理解。为此，我们提出时间门控视频大模型（TG-Vid），通过新颖的时间门控模块（TG）增强时序建模能力。该模块对子模块实施时间门控机制，包括门控空间注意力、门控时序注意力和门控MLP，使模型能更稳健地捕捉视频中的时序信息。在多个时序敏感视频基准（MVBench、TempCompass、NExT-QA）上的广泛评估表明，TG-Vid显著优于现有视频大模型。全面的消融实验验证了性能提升主要归因于时间门控模块的设计。代码已公开于 https://github.com/LaVi-Lab/TG-Vid。
原文摘要 · Abstract (English)
Video Large Language Models (Video LLMs) have achieved impressive performance on video-and-language tasks, such as video question answering. However, most existing Video LLMs neglect temporal information in video data, leading to struggles with temporal-aware video understanding. To address this gap, we propose a Time Gating Video LLM (TG-Vid) designed to enhance temporal modeling through a novel Time Gating module (TG). The TG module employs a time gating mechanism on its sub-modules, comprising gating spatial attention, gating temporal attention, and gating MLP. This architecture enables our model to achieve a robust understanding of temporal information within videos. Extensive evaluation of temporal-sensitive video benchmarks (i.e., MVBench, TempCompass, and NExT-QA) demonstrates that our TG-Vid model significantly outperforms the existing Video LLMs. Further, comprehensive ablation studies validate that the performance gains are attributed to the designs of our TG module. Our code is available at https://github.com/LaVi-Lab/TG-Vid.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。