用光流运动先验提升长视频理解，让大模型更高效处理小时级视频。
Flow4Agent: Long-form Video Understanding via Motion Prior from Optical Flow
- 利用光流运动信息优化时间粒度，分步过滤冗余帧与场景。
- 在三个基准上实现60.4%~71.4%准确率，小时级视频表现显著领先。
- 适合需要长视频分析的场景，如智能监控、影视内容理解。
长视频理解因时空内容高度冗余而面临挑战，尤其受限于多模态大模型（MLLM）的上下文长度。现有方法多依赖语义先验（如CLIP）提取关键信息，但难以有效去除冗余。本文提出Flow4Agent，首次引入光流运动先验来增强基于大模型的长视频理解。该框架通过两个核心模块：时间粒度优化（TGO）自适应重构帧级层次结构，先以粗粒度光流聚类相似视觉内容，再用语义先验剔除无关场景；运动标记剪枝（MTP）则利用细粒度光流信息，对帧内视觉标记进行冗余剪枝。大量实验表明，该方法在多个视频MLLM基准上优于现有方法，尤其在小时级视频任务中表现突出：在Video-MME上达64.7%，MLVU上达71.4%，LongVideoBench上达60.4%。
原文摘要 · Abstract (English)
Long-form video understanding has always been a challenging problem due to the significant redundancy in both temporal and spatial contents. This challenge is further exacerbated by the limited context length of Multimodal Large Language Models (MLLMs). To address this issue, many previous works have attempted to extract key video information, where the "key" is typically semantic-aware and heavily dependent on the CLIP model as prior. In this paper, we propose Flow4Agent, a novel framework that pioneeringly incorporates motion priors from optical flow to facilitate LLM-based long video understanding. Flow4Agent mitigates the redundancy in long videos at both temporal and spatial levels through two core modules: Temporal Granularity Optimization (TGO) adaptively refines framelevel hierarchies, which first leverages coarse flow priors to group similar visual contents and then applies semantic priors to filter out highly irrelevant scene information. Motion Token Pruning (MTP) further refines the intra-frame visual representations, pruning high-redundancy video tokens using fine-grained optical flow information. Extensive experiments demonstrate that our Flow4Agent outperforms existing methods across a wide range of video MLLM benchmarks, especially for hour-level video understanding tasks, achieving 64.7% on Video-MME, 71.4% on MLVU and 60.4% on LongVideoBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。