让大模型精准定位视频事件时间点,告别依赖语言套路的错误预测。
VideoExpert: Augmented LLM for Temporal-Sensitive Video Understanding
- 分设时序与空间专家,分别处理时间定位和内容理解。
- 在多个视频任务上实现更准的时间戳预测,尤其提升长视频精度。
- 适合需要精确时间定位的视频分析场景,如体育回放、事故追溯。
视频理解的核心挑战在于捕捉随时间动态变化的内容。然而,多模态大语言模型在需要标记特定事件发生时刻的时序敏感任务中表现不佳,现有方法要求模型直接生成绝对或相对时间戳,导致模型过度依赖语言模式而非视觉线索。为此,我们提出 VideoExpert,一种适用于多种时序敏感视频任务的通用多模态大模型。受专家概念启发,VideoExpert 集成两个并行模块:时序专家负责建模时间序列与进行时序定位,通过处理高帧率压缩令牌捕捉视频动态变化,并配备轻量级预测头实现精确事件定位;空间专家专注于内容细节分析与指令遵循,处理特殊设计的空间令牌与语言输入,生成相关内容回应。两者通过特定令牌协同工作,确保时序定位与内容生成协调一致。关键的是,两个专家保持独立参数集。通过将时序定位从内容生成中解耦,VideoExpert 有效避免了文本模式偏差。此外,我们引入空间压缩模块,过滤并压缩补丁令牌,同时保留关键信息,为空间专家提供紧凑而丰富的输入。大量实验验证了 VideoExpert 的有效性与通用性。
原文摘要 · Abstract (English)
The core challenge in video understanding lies in perceiving dynamic content changes over time. However, multimodal large language models struggle with temporal-sensitive video tasks, which requires generating timestamps to mark the occurrence of specific events. Existing strategies require MLLMs to generate absolute or relative timestamps directly. We have observed that those MLLMs tend to rely more on language patterns than visual cues when generating timestamps, affecting their performance. To address this problem, we propose VideoExpert, a general-purpose MLLM suitable for several temporal-sensitive video tasks. Inspired by the expert concept, VideoExpert integrates two parallel modules: the Temporal Expert and the Spatial Expert. The Temporal Expert is responsible for modeling time sequences and performing temporal grounding. It processes high-frame-rate yet compressed tokens to capture dynamic variations in videos and includes a lightweight prediction head for precise event localization. The Spatial Expert focuses on content detail analysis and instruction following. It handles specially designed spatial tokens and language input, aiming to generate content-related responses. These two experts collaborate seamlessly via a special token, ensuring coordinated temporal grounding and content generation. Notably, the Temporal and Spatial Experts maintain independent parameter sets. By offloading temporal grounding from content generation, VideoExpert prevents text pattern biases in timestamp predictions. Moreover, we introduce a Spatial Compress module to obtain spatial tokens. This module filters and compresses patch tokens while preserving key information, delivering compact yet detail-rich input for the Spatial Expert. Extensive experiments demonstrate the effectiveness and versatility of the VideoExpert.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。