用固定容量摘要态记忆，解决视频流延迟查询的线索丢失问题。
ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory
- 将远期视频内容压缩为语义空间原型，保持恒定内存占用。
- 在长延迟场景下，准确率比传统方法提升最多12.5个百分点。
- 适合需要低延迟、高吞吐的实时视频理解任务。
流式视频理解（SVU）需在严格限制的GPU内存和查询响应时间预算下，处理异步到达的查询，并持续接收视觉标记流。主要挑战是延迟查询：关键线索可能短暂出现，但在查询到达前已有大量后续更新，导致线索在有限内存中被丢弃或稀释。我们提出ProtoKV，一种恒定足迹的SVU记忆机制，将远期历史表示为固定容量的摘要状态，而非保留原始标记实例。ProtoKV维持一个精确的近窗键值缓存，并将旧内容聚合到带有残差统计信息的语义-空间原型库中。查询时，每个原型通过受限伪标记接口暴露，与标准注意力机制完全兼容。在相同资源预算和相当的查询开销下，ProtoKV在长延迟条件下相较于标记保留基线，准确率最高提升12.5点，且增益随查询延迟增加而扩大。
原文摘要 · Abstract (English)
Streaming video understanding (SVU) must answer queries that arrive asynchronously while visual tokens stream continuously under strict GPU-memory and query-time latency budgets. A key challenge is delayed query: decisive cues may appear briefly, yet many subsequent updates occur before the query arrives, increasing the risk that those cues are evicted or diluted under bounded memory. We propose ProtoKV, a constant-footprint SVU memory that represents far history as a fixed-capacity summary state rather than retaining token instances. ProtoKV keeps an exact near-window KV cache and aggregates older content into a semantic-spatial prototype bank with residual statistics. At query time, each prototype is exposed through a bounded pseudo-token interface that is drop-in compatible with standard attention. Under matched budgets and comparable query-time cost, ProtoKV improves accuracy by up to 12.5 points over token-retention baselines on SVU benchmarks in the long-delay regime, with gains that grow as query delay increases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。