让大模型像侦探一样反复寻找长视频中的关键线索，高效答题。
Video Detective: Seek Critical Clues Recurrently to Answer Question from Long Videos
- 通过迭代处理视频片段，用特殊记忆标记压缩信息，精准定位关键内容。
- 仅用32K上下文长度处理100K视觉标记（1小时视频），耗时2分37秒，显存仅37GB。
- 新数据集GLVC评估模型找关键线索能力，适合研究长视频理解的学者。
长视频问答（LVQA）对多模态大模型构成巨大挑战，因上下文冗长、信息过载，常导致内存消耗过高。现有方法或减少视觉标记数，或扩展上下文长度，但可能遗漏关键信息或计算开销大。事实上，回答问题只需少量关键信息。为此，我们提出一种高效的问答感知记忆机制，使模型能反复搜索关键线索。所提方法VideoDetective通过迭代处理视频子段，每段引入少量特殊记忆标记，实现有目的的信息压缩。同时，将历史记忆不断聚合并更新，供后续子段复用。为更有效评估模型长视频理解能力，我们构建了新数据集GLVC，其包含散布于整段视频中的具体可定位关键线索。实验表明，该方法使上下文限制为32K的模型能高效处理100K视觉标记（约1小时视频，1fps采样），仅需2分37秒和37GB GPU显存。多个长视频基准测试结果证明，本方法能更有效地从海量信息中提取关键线索。
原文摘要 · Abstract (English)
Long Video Question-Answering (LVQA) presents a significant challenge for Multi-modal Large Language Models (MLLMs) due to immense context and overloaded information, which could also lead to prohibitive memory consumption. While existing methods attempt to address these issues by reducing visual tokens or extending model's context length, they may miss useful information or take considerable computation. In fact, when answering given questions, only a small amount of crucial information is required. Therefore, we propose an efficient question-aware memory mechanism, enabling MLLMs to recurrently seek these critical clues. Our approach, named VideoDetective, simplifies this task by iteratively processing video sub-segments. For each sub-segment, a question-aware compression strategy is employed by introducing a few special memory tokens to achieve purposefully compression. This allows models to effectively seek critical clues while reducing visual tokens. Then, due to history context could have a significant impact, we recurrently aggregate and store these memory tokens to update history context, which would be reused for subsequent sub-segments. Furthermore, to more effectively measure model's long video understanding ability, we introduce GLVC (Grounding Long Video Clues), a long video question-answering dataset, which features grounding critical and concrete clues scattered throughout entire videos. Experimental results demonstrate our method enables MLLMs with limited context length of 32K to efficiently process 100K tokens (3600 frames, an hour-long video sampled at 1fps), requiring only 2 minutes and 37GB GPU memory usage. Evaluation results across multiple long video benchmarks illustrate our method can more effectively seek critical clues from massive information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。