用互补视角统一长视频的全局叙事与细节证据,提升理解效果。
Beyond Frame Selection: Rethinking Long-Video Understanding with MLLMs

- 构建时间层次结构,分离全局剧情与局部细节视图
- 在VideoMME上比现有方法高2.9分,验证协同推理有效性
- 适合需要兼顾宏观理解与精准定位的长视频任务
多模态大语言模型在视频理解方面取得显著进展,但受制于上下文长度限制,难以捕捉时间稀疏的关键证据。现有方法通常依赖均匀采样或帧选择,仅侧重广度覆盖或局部相关性,难以兼顾全局叙事与细粒度证据。本文提出VideoRouter(VR),将长视频理解重新定义为协调互补证据视图,而非单一帧子集选择。首先,构建无问題导向的时间层次结构,将视频划分为粗粒度到细粒度的时序一致片段:高层节点捕捉整体剧情演进,低层节点保留关键细节时刻。由此自然形成两种互补视图:全局视图用于覆盖性推理,局部视图用于细节证据恢复。进一步引入验证引导的路由机制,判断哪种视图更被证据支持,并据此选择最终答案。大量实验表明,该设计有效协调了全局与局部推理,在使用LLaVA-Video-7B模型时,于VideoMME基准上超越当前最优帧选择方法2.9分。代码将公开。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved strong progress in video understanding, yet it remains challenging because the token limitation makes MLLMs difficult to capture temporally sparse evidence. Existing methods typically rely on uniform sampling, or frame selection, but these strategies usually optimize either broad temporal coverage or local relevance, making it difficult to preserve both global storyline context and fine-grained evidence. We propose VideoRouter(VR) that rethinks long-video understanding as coordinating complementary evidence views rather than selecting a single subset of frames. It first organizes each video into a question-agnostic temporal hierarchy, which partition the video into coarse-to-fine temporally coherent segments. In this hierarchy, upper-level nodes capture broad storyline context and event progression, while lower-level nodes preserve fine-grained local details and evidence-bearing moments. This naturally gives rise to two complementary views: a global view for coverage-oriented reasoning and a local view for detail-oriented evidence recovery. We further introduce a verification-guided router to determine which view is better supported by the selected evidence and select the final answer. We validate the effectiveness of the proposed design through extensive experiments, showing that the verification-guided router effectively coordinates global and local reasoning, and that, on VideoMME, our method outperforms state-of-the-art frame selection methods by 2.9 points, respectively, under the LLaVA-Video-7B backbone. We will release the code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。