构建分层记忆的智能体,让模型持续理解视频流并主动提取关键信息。
EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use

- 用分层记忆架构:短期捕捉事件边界,长期按事件存档
- 通过动态采样和主动工具使用,在有限上下文里保持长程推理能力
- 适合需要持续理解复杂视频流的研究者或开发者
在线视频理解要求模型在可能无限的视觉流中持续感知并进行长程推理。其核心挑战在于流媒体输入的无界性与多模态大语言模型(MLLM)有限上下文窗口之间的矛盾。现有方法多依赖被动处理,常在维持长程上下文与捕捉细粒度细节之间权衡。为此,我们提出 EventMemAgent,一种基于分层记忆模块的主动在线视频智能体框架。该框架采用双层策略:短期记忆检测事件边界,并通过事件粒度的水库采样在固定长度缓冲区中动态处理视频帧;长期记忆则按事件粒度结构化归档过往观察。此外,我们集成多粒度感知工具包以实现主动、迭代的证据收集,并引入代理强化学习(Agentic RL),端到端将推理与工具使用策略内化为智能体的内在能力。实验表明,EventMemAgent 在在线视频基准上表现优异。代码将发布于:https://github.com/lingcco/EventMemAgent。
原文摘要 · Abstract (English)
Online video understanding requires models to perform continuous perception and long-range reasoning within potentially infinite visual streams. Its fundamental challenge lies in the conflict between the unbounded nature of streaming media input and the limited context window of Multimodal Large Language Models (MLLMs). Current methods primarily rely on passive processing, which often face a trade-off between maintaining long-range context and capturing the fine-grained details necessary for complex tasks. To address this, we introduce EventMemAgent, an active online video agent framework based on a hierarchical memory module. Our framework employs a dual-layer strategy for online videos: short-term memory detects event boundaries and utilizes event-granular reservoir sampling to process streaming video frames within a fixed-length buffer dynamically; long-term memory structuredly archives past observations on an event-by-event basis. Furthermore, we integrate a multi-granular perception toolkit for active, iterative evidence capture and employ Agentic Reinforcement Learning (Agentic RL) to end-to-end internalize reasoning and tool-use strategies into the agent's intrinsic capabilities. Experiments show that EventMemAgent achieves competitive results on online video benchmarks. The code will be released here: https://github.com/lingcco/EventMemAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。