通过复用视频帧间计算,让视频语言模型更快生成嵌入。
Déjà Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reuse
- 设计可学习的帧间计算复用机制,减少冗余推理。
- 在2%误差内实现最高2.64倍加速,显著提升处理效率。
- 适合需要大规模视频分析的系统开发者使用。
近期,视频-语言模型(VideoLMs)展现出强大能力,为灵活的视频查询系统提供了巨大潜力。这些模型通常依赖视觉变换器(ViTs),对每帧独立处理以提取视觉嵌入。然而,大规模视频生成嵌入需对大量帧进行ViT推理,成为实际部署的主要障碍,亟需集成到可扩展的视频数据管理系统中。本文提出Déjà Vu,一个基于计算复用加速ViT型VideoLMs的视频-语言查询引擎。其核心是专为视频任务设计的ReuseViT模型,能学习识别帧间复用机会,在精度与复用之间取得良好平衡。尽管ReuseViT大幅减少计算量,但此节省并未直接转化为GPU性能提升。为此,Déjà Vu引入内存-计算联合压缩技术,将浮点运算量(FLOP)节省转化为实际性能增益。在三个VideoLM任务上的评估显示,Déjà Vu在2%误差范围内,使嵌入生成速度最高提升2.64倍,极大增强了VideoLM在大规模视频分析中的实用性。
原文摘要 · Abstract (English)
Recently, Video-Language Models (VideoLMs) have demonstrated remarkable capabilities, offering significant potential for flexible and powerful video query systems. These models typically rely on Vision Transformers (ViTs), which process video frames individually to extract visual embeddings. However, generating embeddings for large-scale videos requires ViT inferencing across numerous frames, posing a major hurdle to real-world deployment and necessitating solutions for integration into scalable video data management systems. This paper introduces Déjà Vu, a video-language query engine that accelerates ViT-based VideoLMs by reusing computations across consecutive frames. At its core is ReuseViT, a modified ViT model specifically designed for VideoLM tasks, which learns to detect inter-frame reuse opportunities, striking an effective balance between accuracy and reuse. Although ReuseViT significantly reduces computation, these savings do not directly translate into performance gains on GPUs. To overcome this, Déjà Vu integrates memory-compute joint compaction techniques that convert the FLOP savings into tangible performance gains. Evaluations on three VideoLM tasks show that Déjà Vu accelerates embedding generation by up to a 2.64x within a 2% error bound, dramatically enhancing the practicality of VideoLMs for large-scale video analytics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。