提出无需微调的并行编码方法,显著加速长视频理解
PEVLM: Parallel Encoding for Vision-Language Models
- 将视频分块+共享汇聚块,降低注意力复杂度
- 速度提升7.47倍,端到端延迟减少40%
- 适合对延迟敏感的实时长视频应用
视觉语言模型在多模态理解与生成任务中表现优异,但在处理长视频时受限于标准注意力机制的二次复杂度。本文提出无需微调的并行编码方法PEVLM,将输入视频划分为含共享汇聚块的上下文块,并保留序列位置嵌入,使注意力权重分布与全注意力对齐。该设计将注意力复杂度从O((T×N)²)降至O(T×N),其中T为帧数,N为每帧标记数,且不损失精度。大量实验表明,PEVLM在多个先进模型与基准上均优于现有并行编码方法,在注意力计算上实现最高7.47倍加速,端到端延迟降低40%。令人惊讶的是,其在部分场景下甚至超越全注意力性能;在严格延迟约束下,准确率从23.26%提升至61.03%。结果证明PEVLM在低延迟长上下文视频理解中的有效性,为实际应用提供了有力方案。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have demonstrated strong capabilities in multimodal understanding and generation tasks. However, their application to long video understanding remains hindered by the quadratic complexity of standard attention mechanisms. In this work, we introduce \textbf{PEVLM}, a fine-tuning-free parallel encoding method designed to enhance the prefilling efficiency of VLMs in long video scenarios. PEVLM partitions the input video into context blocks with a shared sink block, while preserving sequential position embeddings to align the attention weight distribution with that of Full-Attention. This design reduces attention complexity from $O((T \times N)^2)$ to $O(T \times N)$ where $T$ is the number of frames and $N$ the number of tokens per frame, without sacrificing accuracy. Extensive experiments across multiple state-of-the-art models and benchmarks demonstrate that PEVLM consistently outperforms existing parallel encoding approaches, achieving up to \textbf{7.47x} speedup in attention computation and reducing end-to-end latency by \textbf{40\%}. Remarkably, PEVLM not only maintains high accuracy, but in some settings even surpasses Full-Attention performance. Under strict latency constraints, it achieves substantial gains, improving accuracy from \textbf{23.26\%} to \textbf{61.03\%}. These results underscore the effectiveness of PEVLM for low-latency, long-context video understanding, making it a promising solution for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。