让长视频理解更连贯:用事件记忆替代碎片帧,提升问答准确性。
Video-EM: Event-Centric Episodic Memory for Long-Form Video Understanding
- 以事件为中心构建记忆,将关键帧聚合成时间连贯的叙事单元。
- 通过自反思机制去除冗余,生成带时空索引的紧凑事件时间线。
- 无需训练即可适配现有视频大模型,特别适合长视频问答任务。
视频大语言模型在视频理解方面表现强劲,但处理长视频时受限于有限的上下文窗口。现有方法常通过检索或摘要将长视频压缩为少数代表性帧,但大多孤立评分,假设帧级显著性足以支持下游推理,导致选择冗余、时序证据断裂、叙事基础薄弱。本文提出Video-EM,一种无需训练的事件中心情景记忆框架,将长视频问答重构为事件构建与记忆精炼的过程。不将检索到的关键帧视为独立视觉,而是利用大模型作为主动记忆代理,协调现成工具:首先通过多粒度语义匹配定位查询相关时刻,再分组分割为时间连贯的事件,并将每个事件编码为包含明确时间索引与时空线索的可追溯情景记忆(涵盖何时、何地、何事及涉及实体)。为抑制上游信号不完美带来的冗余和噪声,Video-EM引入基于推理的自反思循环,迭代验证证据充分性与跨事件一致性,消除冗余并自适应调整事件粒度。最终输出一个紧凑可靠的事件时间线——可直接被现有视频大模型使用,无需额外训练或架构改动。
原文摘要 · Abstract (English)
Video Large Language Models (Video-LLMs) have shown strong video understanding, yet their application to long-form videos remains constrained by limited context windows. A common workaround is to compress long videos into a handful of representative frames via retrieval or summarization. However, most existing pipelines score frames in isolation, implicitly assuming that frame-level saliency is sufficient for downstream reasoning. This often yields redundant selections, fragmented temporal evidence, and weakened narrative grounding for long-form video question answering. We present \textbf{Video-EM}, a training-free, event-centric episodic memory framework that reframes long-form VideoQA as \emph{episodic event construction} followed by \emph{memory refinement}. Instead of treating retrieved keyframes as independent visuals, Video-EM employs an LLM as an active memory agent to orchestrate off-the-shelf tools: it first localizes query-relevant moments via multi-grained semantic matching, then groups and segments them into temporally coherent events, and finally encodes each event as a grounded episodic memory with explicit temporal indices and spatio-temporal cues (capturing \emph{when}, \emph{where}, \emph{what}, and involved entities). To further suppress verbosity and noise from imperfect upstream signals, Video-EM integrates a reasoning-driven self-reflection loop that iteratively verifies evidence sufficiency and cross-event consistency, removes redundancy, and adaptively adjusts event granularity. The outcome is a compact yet reliable \emph{event timeline} -- a minimal but sufficient episodic memory set that can be directly consumed by existing Video-LLMs without additional training or architectural changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。