用记忆机制增强视频理解，让模型更准更省资源。
Memory-enhanced Retrieval Augmentation for Long Video Understanding
- 模拟人脑记忆，分步记忆、推理、检索关键片段
- 在多个基准上超越现有模型，准确率显著提升
- 适合需要长视频分析的科研与工业场景
高效理解长视频仍是计算机视觉中的挑战。当前长上下文视觉语言模型（LVLMs）因压缩和暴力降采样导致信息丢失。尽管检索增强生成（RAG）方法缓解此问题，但受限于显式查询依赖。为此，我们提出名为 MemVid 的新型记忆增强 RAG 方法，受人类认知记忆启发。该方法包含四步：1）记忆视频整体信息；2）基于记忆推理任务所需信息；3）根据需求检索关键片段；4）聚焦检索结果生成最终答案。为增强记忆引导推理能力并实现最优端到端性能，我们设计课程学习策略：先在标注良好的推理结果上进行监督学习，再通过强化学习逐步探索并强化更合理的推理路径。我们在 MLVU、VideoMME 和 LVBench 等主流长视频理解基准上进行评估，实验表明，MemVid 在效率和效果上均优于现有 LVLMs 与 RAG 方法。
原文摘要 · Abstract (English)
Efficient long-video understanding~(LVU) remains a challenging task in computer vision. Current long-context vision-language models~(LVLMs) suffer from information loss due to compression and brute-force downsampling. While retrieval-augmented generation (RAG) methods mitigate this issue, their applicability is limited due to explicit query dependency. To overcome this challenge, we introduce a novel memory-enhanced RAG-based approach called MemVid, which is inspired by the cognitive memory of human beings. Our approach operates in four basic steps: 1) memorizing holistic video information, 2) reasoning about the task's information needs based on memory, 3) retrieving critical moments based on the information needs, and 4) focusing on the retrieved moments to produce the final answer. To enhance the system's memory-grounded reasoning capabilities while achieving optimal end-to-end performance, we propose a curriculum learning strategy. This approach begins with supervised learning on well-annotated reasoning results, then progressively explores and reinforces more plausible reasoning outcomes through reinforcement learning. We perform extensive evaluations on popular LVU benchmarks, including MLVU, VideoMME and LVBench. In our experiments, MemVid demonstrates superior efficiency and effectiveness compared to both LVLMs and RAG methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。