arXiv:2609.04131cs.CV2026-09

让视频理解模型学会逐步内化历史信息,提升持续推理能力。

Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding

论文配图:Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding
图 1 · 摘自论文原文
  • 将历史视觉信息从外部检索转为内部渐进式融合,构建动态记忆。
  • 在多个视频基准上达到新最好效果,显著提升在线与离线任务性能。
  • 适合需要长期连续推理的流式视频应用,如实时监控与交互系统。

流式视频理解要求多模态大语言模型在严格因果约束和有限记忆条件下,持续处理视觉输入并响应用户查询。现有方法通常将历史观测压缩至外部记忆库,并通过检索相关证据作为额外视觉上下文。尽管有效,这种存取范式使历史证据始终处于外部上下文,难以内化为可演化、紧凑的潜在记忆,从而限制持续推理能力。为此,我们提出 LatentStream,一种渐进式潜在工作记忆框架,推动流式记忆从“存储-检索”转向“检索-内化”。具体包括三个协同组件:首先,无查询依赖的分层流式记忆通过詹克斯引导的自适应整合,在固定内存预算下组织短、中、长期视觉历史;当查询到达时,分层潜在记忆演化机制为潜在记忆令牌赋予逐级扩展的记忆感知范围,使其迭代地从对应范围内检索历史证据并内化为紧凑固定长度的潜在表示;最后,渐进置信度引导的潜在记忆优化构建组级别预测熵的分层奖励,联合优化潜在记忆令牌与检索证据,促进推理置信度持续提升。大量实验表明,LatentStream 在现有在线与离线视频基准上均取得新最佳性能。

原文摘要 · Abstract (English)

Streaming video understanding requires multimodal large language models (MLLMs) to process continuous visual inputs and respond to user queries under strict causality and bounded memory. Existing approaches typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. Though effective, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning. To bridge this gap, we introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize. Specifically, LatentStream comprises three coordinated components. First, Query-agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels under a fixed memory budget through Jenks-guided adaptive consolidation. Once a query arrives, Hierarchical Latent Memory Evolution equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling them to iteratively retrieve historical evidence from their corresponding scopes and internalize it into a compact, fixed-length latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization constructs a hierarchical progression reward from group-wise predictive entropy and jointly refines the latent memory tokens and retrieved evidence, encouraging increasingly confident streaming reasoning. Extensive experiments demonstrate that LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.

视频理解流式处理记忆演化多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。