为流式视频设计聚焦语义记忆,高效保留关键信息。
FOLIO: Focused Semantic Memory for Streaming Video Understanding

- 按片段动态聚焦,区分重要实体与背景上下文
- 在OVO-Bench上达82.0/69.1感知与回溯准确率
- 适合需持续理解长视频流的实时应用
在线流式视频理解中,视频持续输入,查询可能随时发生。由于帧流无界增长,系统必须持续压缩并保留已观察视频前缀的信息,同时未来帧和查询未知。核心挑战在于决定保留什么信息以及如何组织历史记录:随着历史增长,内存开销上升,大量冗余视觉细节被保留,而后续查询通常依赖特定实体、动作及其时间变化。为此,我们提出FOLIO——一种无需训练的聚焦语义记忆系统,在保持周围上下文紧凑的同时,以更高精度记录流中重要部分。当流数据到达时,FOLIO基于动态关注状态在段级别更新记忆,结合短期视觉缓冲区与围绕观测实体组织的长期语义记忆,并链接至视觉证据缓存。查询时,轻量级混合检索结合结构化记忆的直接匹配与语义查询扩展。FOLIO在OVO-Bench上以Qwen3-VL-8B达到82.0/69.1的感知/回溯准确率,在StreamingBench上实现74.5的整体准确率,同时显著降低维护流式记忆的成本,仅对关注实体保留详细记录,背景上下文则紧凑存储。
原文摘要 · Abstract (English)
In online streaming video understanding, a video stream continues to arrive and queries may be issued at any time. Because streaming frames grow without bound, the system must continuously compress and retain information from the observed video prefix while future frames and future queries remain unknown. The core challenge is deciding what information to retain and how to organize the maintained history: as this history grows with the stream, memory cost increases and many redundant visual details are retained, whereas later queries often depend on specific entities, actions, and their temporal changes. To address this challenge, we introduce FOLIO, a training-free focused semantic memory system that records important parts of the stream in higher detail while keeping surrounding context compact. As the stream arrives, FOLIO updates memory at the segment level, guided by a dynamic focus state, combining a short-term visual buffer with a long-term semantic memory organized around observed entities and linked to a visual-evidence cache. At query time, lightweight hybrid retrieval combines direct matching over the structured memory with semantic query expansion. FOLIO achieves state-of-the-art performance, reaching 82.0/69.1 Perception/Backward accuracy on OVO-Bench with Qwen3-VL-8B and 74.5 overall accuracy on StreamingBench, while substantially reducing the cost of maintaining streaming memory by reserving detailed records for focused entities and storing surrounding context compactly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。