arXiv:2505.16175cs.CVcs.AI2025-05被引 9

提出QuickVideo系统,让长视频理解实时运行。

QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Design

  • 用并行解码和缓存压缩,加速视频处理与推理
  • 在长视频上减少约1分钟推理时间,降低内存占用
  • 适合部署在资源有限的设备上做实时视频分析

长视频理解在视频监控、会议摘要、教育讲座分析和体育直播等实际应用中至关重要。然而,由于两大瓶颈:1)逐帧解码耗时长达一分钟(对小时级视频),2)大模型推理需预填充数百万令牌,导致高延迟和高内存消耗,现有VideoLLM难以满足实时需求。为此,我们提出QuickVideo,一种系统-算法协同设计方法,显著加速长视频理解,支持实时下游应用。其核心包括:快速解码器(QuickDecoder),基于CPU并行处理关键帧对齐区间,提速2-3倍;快速预填充(QuickPrefill),通过KV缓存剪枝减少内存使用,支持更多帧;以及解码与推理重叠机制,实现计算流水线优化。三者结合使长视频推理时间减少约1分钟,可在有限硬件上实现可扩展、高质量的视频理解。实验表明,QuickVideo适用于不同长度和采样率,使长视频处理真正可行。

原文摘要 · Abstract (English)

Long-video understanding has emerged as a crucial capability in real-world applications such as video surveillance, meeting summarization, educational lecture analysis, and sports broadcasting. However, it remains computationally prohibitive for VideoLLMs, primarily due to two bottlenecks: 1) sequential video decoding, the process of converting the raw bit stream to RGB frames can take up to a minute for hour-long video inputs, and 2) costly prefilling of up to several million tokens for LLM inference, resulting in high latency and memory use. To address these challenges, we propose QuickVideo, a system-algorithm co-design that substantially accelerates long-video understanding to support real-time downstream applications. It comprises three key innovations: QuickDecoder, a parallelized CPU-based video decoder that achieves 2-3 times speedup by splitting videos into keyframe-aligned intervals processed concurrently; QuickPrefill, a memory-efficient prefilling method using KV-cache pruning to support more frames with less GPU memory; and an overlapping scheme that overlaps CPU video decoding with GPU inference. Together, these components infernece time reduce by a minute on long video inputs, enabling scalable, high-quality video understanding even on limited hardware. Experiments show that QuickVideo generalizes across durations and sampling rates, making long video processing feasible in practice.

视频理解系统优化实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。