用MapReduce思想处理长视频，提升理解准确率
MR. Video: "MapReduce" is the Principle for Long Video Understanding
- 分段感知+全局聚合，突破上下文长度限制
- 在LVBench上比顶尖模型高10%以上准确率
- 适合需要长视频精细分析的场景
我们提出MR. Video，一种基于MapReduce原理的长视频理解框架：先将视频切分为短片段独立感知（映射），再统一整合所有片段信息（归约）。相比序列到序列的视觉语言模型，该方法可无约束地进行细粒度短片段感知；相较于依赖顺序关键片段选择的现有视频智能体，其并行感知更简单且可扩展。归约步骤实现更全面的上下文聚合与推理，优于显式的关键片段检索。该原则适用于视觉语言模型与视频智能体，我们使用大语言模型智能体验证其有效性。实际应用中包含两个阶段：(A) 对短片段生成字幕（映射），再统一标准化重复角色与物体名称（归约）；(B) 针对用户问题，分别分析各片段信息（映射），再整合成最终答案（归约）。在挑战性数据集LVBench上，MR. Video相比最优视觉语言模型与视频智能体，准确率提升超10%。代码已开源。
原文摘要 · Abstract (English)
We propose MR. Video, an agentic long video understanding framework that demonstrates the simple yet effective MapReduce principle for processing long videos: (1) Map: independently and densely perceiving short video clips, and (2) Reduce: jointly aggregating information from all clips. Compared with sequence-to-sequence vision-language models (VLMs), MR. Video performs detailed short video perception without being limited by context length. Compared with existing video agents that typically rely on sequential key segment selection, the Map operation enables simpler and more scalable sequence parallel perception of short video segments. Its Reduce step allows for more comprehensive context aggregation and reasoning, surpassing explicit key segment retrieval. This MapReduce principle is applicable to both VLMs and video agents, and we use LLM agents to validate its effectiveness. In practice, MR. Video employs two MapReduce stages: (A) Captioning: generating captions for short video clips (map), then standardizing repeated characters and objects into shared names (reduce); (B) Analysis: for each user question, analyzing relevant information from individual short videos (map), and integrating them into a final answer (reduce). MR. Video achieves over 10% accuracy improvement on the challenging LVBench compared to state-of-the-art VLMs and video agents. Code is available at: https://github.com/ziqipang/MR-Video
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。