用分层精炼方法，从长视频中高效提取关键帧提升理解精度
Progressive Video Condensation with MLLM Agent for Long-form Video Understanding
- 分三步逐步缩小范围：定位相关片段→筛选重要短片段→精确定位关键帧
- 零样本下在EgoSchema等3个数据集上达到69.3%~80.5%准确率
- 仅需少量帧即可完成推理，适合计算资源受限的视频理解任务
长视频理解需在有限算力下从长序列中提取查询相关的信息。现有文本优先的LLM流程会丢失细粒度视觉线索，而基于视频的多模态大语言模型（MLLM）虽能保留视觉细节，但帧数需求高、计算成本大。本文提出ProVCA，一种渐进式视频压缩代理，通过多粒度迭代定位关键帧。首先使用段落定位模块识别与查询相关的视频段，再通过片段选择模块基于相似性筛选重要短片段，最后通过关键帧精炼模块在这些片段中精准定位具体关键帧。通过从粗到细逐步缩小范围，ProVCA为MLLM推理选出少量关键帧。在EgoSchema、NExT-QA和IntentQA上实现69.3%、80.5%和77.7%的零样本最优准确率，且使用帧数少于以往无训练方法。
原文摘要 · Abstract (English)
Understanding long videos requires extracting query-relevant information from long sequences under tight compute budgets. Existing text-then-LLM pipelines lose fine-grained visual cues, while video-based multimodal large language models (MLLMs) can keep visual details but are too frame-hungry and computationally expensive. In this work, we aim to harness MLLMs for efficient video understanding. We propose ProVCA, a progressive video condensation agent that iteratively locates key video frames at multiple granularities. ProVCA first adopts a segment localization module to identify the video segment relevant to the query, then a snippet selection module to select important snippets based on similarity, and finally a keyframe refinement module to pinpoint specific keyframes in those snippets. By progressively narrowing the scope from coarse segments to fine frames, ProVCA identifies a small set of keyframes for MLLM-based reasoning. ProVCA achieves state-of-the-art zero-shot accuracies of 69.3\% on EgoSchema, 80.5\% on NExT-QA, and 77.7\% on IntentQA, while using fewer frames than previous training-free methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。