通过自适应记忆管理，让模型高效理解超长视频内容。
VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Management
- 将超长视频理解视为序列生成任务，动态维护关键信息记忆
- 在多个基准上超越现有开源模型，显著提升长视频理解性能
- 适合需要处理长时间视频的场景，如智能监控、纪录片分析
超长视频理解仍是开放挑战，因现有视觉语言模型受限于上下文长度和长期记忆效率。尽管近期工作尝试构建外部知识库与检索增强生成（RAG）系统，但带来巨大存储与计算开销。本文提出VideoMem，首次将长视频理解建模为序列生成任务，通过自适应记忆管理实现全局记忆缓冲区的动态更新，保留关键信息并丢弃冗余内容。为高效训练视觉语言模型，VideoMem集成渐进分组相对策略优化（PRPO）算法，包含两个核心模块：渐进状态传播（PSP）自适应保留有效状态并传递至下一阶段，逐步缩小探索空间；时间级联奖励（TCR）缓解奖励稀疏问题，提升样本利用率并加速收敛。大量实验表明，VideoMem在多个超长视频理解基准上显著优于现有开源模型。
原文摘要 · Abstract (English)
Ultra long video understanding remains an open challenge, as existing vision language models (VLMs) falter on such content due to limited context length and inefficient long term memory retention. To address this, recent works have attempted to construct external knowledge bases and corresponding retrieval agumented generation (RAG) systems, yet these incur enormous storage and computational overhead. In this paper, we propose VideoMem, a novel framework that pioneers models long video understanding as a sequential generation task via adaptive memory management. Specifically, VideoMem dynamically updates a global memory buffer, which adaptively retains critical information while discarding redundant content across the video timeline. To efficiently train VLMs for such long-term tasks, VideoMem integrates the Progressive Grouped Relative Policy Optimization (PRPO) algorithm, equipped with two core modules: Progressive State Propagation (PSP) adaptively retains valid current states, propagates them to the next rollout step, and gradually narrows the model exploration space. Temporal Cascading Reward (TCR) further alleviates reward sparsity, improving sample utilization and accelerating convergence. Extensive experiments demonstrate that VideoMem significantly outperforms existing open-source models across diverse benchmarks for ultra-long video understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。