动态分粒度模块化框架,让大模型更准理解长视频中的多个关键事件。
Modularized Dynamic-Granularity Video LLM for Multi-Event Long Video Understanding

- 构建闭环机制,动态划分视频细粒度编码,精准定位关键片段。
- 在多事件长视频基准上超越现有方法,性能提升显著。
- 适合需要细粒度视频理解与推理的复杂任务场景。
视频大语言模型在各类视频理解任务中已取得显著进展,但长视频场景仍具挑战,主要源于视觉标记预算有限与捕捉多个关键事件之间的矛盾。现有方法通常分两阶段处理:先选取关键帧,再进行细节感知,存在自适应能力不足、缺乏自我修正机制等问题,导致建模不可靠。为此,我们提出 MoD-VLLM——一种新型模块化动态粒度视频大模型框架,用于多事件长视频理解。该框架通过正负视频段定位模块与模块化动态粒度反思模块形成闭环,迭代地定位与问题相关的视频片段。定位模块指导视频大模型区分相关与无关片段;反思模块采用模块化调度器,对正向片段采用细粒度编码以捕捉细节,对负向片段采用粗粒度编码以保留全局上下文。此外,我们设计了一种动态粒度强化学习策略,使模型联合学习最优定位策略与动态粒度视觉表示。同时,我们构建了 MEventBench——一个面向复杂长视频推理的多事件长视频基准。大量实验在多个长视频理解基准及 MEventBench 上表明,MoD-VLLM 显著优于当前最优基线。
原文摘要 · Abstract (English)
Video Large Language Models (Video LLMs) have made significant advancements in various video understanding tasks. However, long-video scenarios remain challenging due to the tension between limited visual token budgets and the need to capture multiple key events. Existing approaches typically process long videos in two stages, i.e., i) select keyframes and ii) perform detailed perception, which exhibit limitations: they lack a modular mechanism for adaptive capacity allocation and self-correction, resulting in unreliable modeling. To tackle these challenges, we propose MoD-VLLM, a novel Modularized Dynamic-Granularity Video LLM framework for multi-event long video understanding, which unifies temporal grounding and semantic understanding iteratively and self-reflectively. Specifically, we propose a Positive-Negative Video Segments Grounding module and a Modularized Dynamic-Granularity Reflection module, which form a closed loop to progressively localize the question-related video segments. The grounding module instructs a Video LLM to distinguish relevant from irrelevant video segments based on the video question. The reflection module employs a modularized scheduler that dynamically selects fine-grained encoding for relevant positive segments to capture detailed perception and coarse-grained encoding for negative segments to maintain global context. We further propose a dynamic-granularity reinforcement learning strategy, allowing MoD-VLLM to learn optimal grounding policies and dynamic granularity visual representation jointly. Moreover, we propose MEventBench, a challenging Multi-Event Long Video Benchmark for complex long video reasoning. Extensive experiments on several long video understanding benchmarks and our MEventBench demonstrate that MoD-VLLM significantly outperforms state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。