用结构化文本记忆实现流式视频长期理解,无需训练即可高效检索。
Dynamic Hub-and-Spoke Memory for Streaming Video Understanding

- 将历史视频片段转为带类型的文本记忆,以实体为中心构建枢纽-分支结构。
- 动态检索与问题相关的记忆子集,结合近期视觉帧进行答案预测。
- 无需训练,兼容多种模型,在流式和长视频任务上显著优于现有方法。
流式视频理解需在持续增长的视觉流中任意时间回答问题。核心挑战在于如何紧凑地记忆长时历史,同时有效检索与问题相关的内容。我们提出无训练的动态枢纽-分支记忆(D-HSM)框架:将选定的历史视频段落转化为类型化的文本观测,并存储于以实体为中心的枢纽-分支记忆结构中,实体为枢纽,关联证据为分支。回答问题时,D-HSM动态检索紧凑的问题感知记忆子集,通过枢纽-分支链接扩展,并与最近视觉窗口结合,用于冻结视觉语言模型(VLM)的答案预测。在流式与长视频基准上的大量实验表明,D-HSM持续且显著提升VLM主干模型性能,优于其他先进在线与离线视频理解基线。
原文摘要 · Abstract (English)
Streaming video understanding requires answering questions at arbitrary times over a continuously growing visual stream. The central challenge is to compactly remember long-range history while effectively retrieving question-relevant evidence. We propose Dynamic Hub-and-Spoke Memory (D-HSM), a training-free framework that represents distant history as structured textual memory while preserving the recent frames as visual tokens for fine-grained perception. Specifically, D-HSM turns selected historical video chunks into typed textual observations and stores them in an entity-centered hub-and-spoke memory, with entities as hubs and related evidence as spokes. When answering a question, D-HSM dynamically retrieves a compact question-aware memory subset, expands it through hub-and-spoke links, and combines it with the recent visual window for frozen-VLM answer prediction. Extensive experiments on both streaming and long video benchmarks show that D-HSM consistently and substantially improves VLM backbones and outperforms other state-of-the-art online and offline video understanding baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。