用压缩令牌实现每小时视频仅占22倍内存，实时处理流程视频。
Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding
- 用文本摘要和视觉特征混合缓存，大幅减少长时视频的令牌数量。
- 在4个数据集上6项任务达新基准，支持每秒10帧流式推理。
- 仅需2GB显存，适合部署在资源受限设备上。
我们提出ProVideLLM，一种端到端的实时流程视频理解框架。该框架采用多模态缓存机制，存储两类令牌：语义化文本令牌（对长期观察进行压缩摘要）与经DETR-QFormer编码的视觉令牌（捕捉短期观察的细粒度信息）。此设计将一小时长时观察的令牌数量减少22倍，同时有效保留当前细节。通过交错存储这些令牌，ProVideLLM实现内存与计算量随视频长度近似线性增长，支持每秒10帧的流式推理和每秒25帧的流式对话，仅需2GB GPU显存。此外，ProVideLLM在四个数据集上的六个流程类任务中取得新最佳性能。
原文摘要 · Abstract (English)
We introduce ProVideLLM, an end-to-end framework for real-time procedural video understanding. ProVideLLM integrates a multimodal cache configured to store two types of tokens - verbalized text tokens, which provide compressed textual summaries of long-term observations, and visual tokens, encoded with DETR-QFormer to capture fine-grained details from short-term observations. This design reduces token count by 22x over existing methods in representing one hour of long-term observations while effectively encoding fine-granularity of the present. By interleaving these tokens in our multimodal cache, ProVideLLM ensures sub-linear scaling of memory and compute with video length, enabling per-frame streaming inference at 10 FPS and streaming dialogue at 25 FPS, with a minimal 2GB GPU memory footprint. ProVideLLM also sets new state-of-the-art results on six procedural tasks across four datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。