用固定容量状态记忆帧间信息，实现长视频理解的线性加速
Linear Scaling Video VLMs for Long Video Understanding
- 引入StateKV，在推理时用可更新的状态缓存跨帧上下文
- 在三个数据集上逼近全自注意力性能，且优于滑动窗口方法
- 无需微调或改架构，适合部署于长视频流任务
视频视觉语言模型（VLMs）在长时序和流式场景中应用日益广泛，但多数视频编码器仍依赖时空自注意力机制，导致计算量和延迟随帧数呈二次增长。现有高效方法虽提升可扩展性，但常因过度丢帧或粗略注意力近似而损失精度。本文提出StateKV，一种推理时方法：通过固定容量、基于重要性的循环状态，携带跨帧上下文；同时辅以每帧完整缓存用于解码。在三个长视频基准测试中，覆盖七种不同规模模型与三类架构，StateKV性能接近全自注意力，持续优于主流滑动窗口或基于时效性的流式近似方法，且无需微调或结构修改。该方法还显著降低视频预填充阶段的计算量（以FLOPs衡量），使在固定算力预算下运行更大模型成为可能，从而获得更高准确率。结果表明，StateKV为可扩展长视频理解提供了实用路径。
原文摘要 · Abstract (English)
Video vision-language models (VLMs) are increasingly used in long-horizon and streaming settings, yet most video encoders still rely on spatiotemporal self-attention, causing compute and latency to grow quadratically with the number of frames. Existing efficiency methods improve scalability but often lose accuracy relative to full self-attention, for example through aggressive frame/token dropping or coarse attention approximations. We introduce StateKV, an inference-time method that adapts pretrained long-video VLMs to linear-time video prefill by carrying cross-frame context in a fixed-capacity, importance-based recurrent state, paired with a second full per-frame cache used for decoding. Across three long-video benchmarks and seven models spanning three families and multiple scales, StateKV remains close to full self-attention and consistently outperforms dominant sliding-window / recency-based streaming approximations, without fine-tuning or architectural changes. StateKV also reduces video-prefill cost measured FLOPs, enabling stronger accuracy at a fixed compute budget by running larger models. These results suggest a practical step toward scalable long-video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。