用视频编码器的运动向量和残差，大幅降低视频语言模型计算量。
CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling
- 利用视频编码中的运动向量与残差，替代完整图像编码
- 推理速度提升86%，令牌使用减少93%
- 在14个任务上性能不降反升，适合长视频理解场景
视频语言模型（VideoLMs）使AI系统能够理解视频中的时序动态。为适应最大上下文窗口限制，现有方法采用关键帧采样，但常因时间覆盖稀疏而遗漏宏观事件和微观细节。此外，每帧处理完整图像及其标记带来巨大计算开销。本文提出利用视频编码器原生具备的冗余与稀疏性特征（即运动向量与残差），避免对多数帧进行昂贵的全图编码。我们设计轻量级Transformer编码器聚合这些编码器原始数据，并通过预训练策略使编码表示与图像编码器嵌入对齐，加速端到端微调收敛。所提方法CoPE-VideoLM相较标准VideoLM，将首次输出时间缩短86%，令牌使用减少93%。通过调整关键帧与编码器原始数据密度，我们在14个多样化的视频理解基准上保持或超越原有性能，涵盖通用问答、时序与运动推理、长视频理解及空间场景理解等任务。
原文摘要 · Abstract (English)
Video Language Models (VideoLMs) enable AI systems to understand temporal dynamics in videos. To fit within the maximum context window constraint, current methods use keyframe sampling which often misses both macro-level events and micro-level details due to the sparse temporal coverage. Furthermore, processing full images and their tokens for each frame incurs substantial computational overhead. We address these limitations by leveraging video codec primitives (specifically motion vectors and residuals) which natively encode video redundancy and sparsity without requiring expensive full-image encoding for most frames. To this end, we introduce lightweight transformer-based encoders that aggregate codec primitives and align their representations with image encoder embeddings through a pre-training strategy that accelerates convergence during end-to-end fine-tuning. Our approach, CoPE-VideoLM, reduces the time-to-first-token by up to 86% and token usage by up to 93% compared to standard VideoLMs. Moreover, by varying the keyframe and codec primitive densities we maintain or exceed performance on 14 diverse video understanding benchmarks spanning general question answering, temporal and motion reasoning, long-form understanding, and spatial scene understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。