将长视频切分为带主题的语义单元,提升问答准确率。
MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering

- 用自读机制分割视频为事件单元,生成事件概要与代表性帧
- 在相同视觉令牌预算下,最高提升21.8%问答准确率
- 适合长视频理解、多模态问答任务的研究者使用
长视频问答对视觉语言模型(VLMs)仍是挑战,因相关证据稀疏、短暂且分散于长视频中。现有基于帧的方法通过均匀采样、查询感知选帧、视觉令牌压缩和自适应分辨率策略提升效率,但仍依赖孤立碎片化帧作为基本证据单元,限制了模型捕捉连贯事件语义的能力。为此,我们提出MemoryCard,一种基于视频记忆的增强框架,将长视频组织为自包含的Memory Cards。具体地,MemoryCard首先对视频与对齐话语进行自读,将视频分割为语义连贯的单元,每个单元对应一个独立主题或事件。针对每个单元,生成事件级视频概要并选取代表性视觉时刻,最终渲染为统一的Memory Cards用于检索与问答。实验表明,MemoryCard在相当的视觉令牌预算下持续提升长视频问答性能,准确率最高相对提升21.8%。代码已开源:https://github.com/NEUIR/MemoryCard。
原文摘要 · Abstract (English)
Long-video question answering remains challenging for Vision-Language Models (VLMs), as answer-relevant evidence is often sparse, transient, and temporally dispersed across lengthy video contexts. Existing frame-centric approaches improve efficiency through uniform sampling, query-aware frame selection, visual-token compression, and adaptive resolution strategies. However, they still rely on isolated and fragmented frames as the fundamental evidence units, limiting VLMs' ability to effectively capture coherent event-level semantics. To address this limitation, we propose MemoryCard, a video-memory-based augmentation framework that organizes long videos into self-contained Memory Cards. Specifically, MemoryCard first performs a self-reading process over videos and aligned utterances to segment the video into semantically coherent units, each corresponding to a distinct topic or event. For each unit, it generates an event-level video gist and selects representative visual moments, which are then rendered into unified Memory Cards for retrieval and question answering. Experimental results demonstrate that MemoryCard consistently improves long-video QA performance under comparable visual-token budgets, achieving up to a 21.8% relative improvement in accuracy. All code is available at https://github.com/NEUIR/MemoryCard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。