扩大视频记忆令牌预算，提升长时视频理解精度。
Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory
- 动态调整缓存策略，减少冗余并保留时空细节。
- 在CG-Bench、LVBench等数据集上提升8%以上。
- 无需训练即可增强检索，适合长视频分析场景。
流式视频理解需模型持续编码、存储并检索视频流中的信息以支持准确的视频问答。现有先进方法依赖键值缓存累积帧级信息，但每帧令牌数量有限，导致细粒度视觉细节丢失。本文提出扩大令牌预算，实现更精细的时空理解与推理。我们发现当前方法难以处理密集流：特征编码使查询-帧相似度随时间上升，偏向后期帧。为此，提出自适应选择策略，在减少令牌冗余的同时保留局部时空信息。进一步设计无需训练的检索混合专家模型，利用外部模型更精准识别相关帧。所提方法MemStream在CG-Bench、LVBench和VideoMME（Long）上分别较ReKV + Qwen2.5-VL-7B提升+8.0%、+8.5%、+2.4%。
原文摘要 · Abstract (English)
Streaming video understanding requires models to robustly encode, store, and retrieve information from a continuous video stream to support accurate video question answering (VQA). Existing state-of-the-art approaches rely on key-value caching to accumulate frame-level information over time, but use a limited number of tokens per frame, leading to the loss of fine-grained visual details. In this work, we propose scaling the token budget to enable more granular spatiotemporal understanding and reasoning. First, we find that current methods are ill-equipped to handle dense streams: their feature encoding causes query-frame similarity scores to increase over time, biasing retrieval toward later frames. To address this, we introduce an adaptive selection strategy that reduces token redundancy while preserving local spatiotemporal information. We further propose a training-free retrieval mixture-of-experts that leverages external models to better identify relevant frames. Our method, MemStream, achieves +8.0% on CG-Bench, +8.5% on LVBench, and +2.4% on VideoMME (Long) over ReKV with Qwen2.5-VL-7B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。