arXiv:2504.12513cs.CVcs.AI2025-04CVPR

AdaVid让视频模型能根据设备算力自动调整计算量,兼顾效率与性能。

AdaVid: Adaptive Video-Language Pretraining

  • 采用可变嵌入维度的自适应变换器,按需调节计算量
  • 用一半算力达到标准模型性能,同等算力下表现更优
  • 支持长视频处理,适合边缘设备部署

对比视频-语言预训练在学习丰富且鲁棒的视频表征方面取得了显著成功。然而,由于计算需求高,将此类视频编码器部署在算力受限的边缘设备上仍具挑战性。此外,现有模型通常仅能处理短视频片段,一般限制在4至64帧。本文提出AdaVid,一种灵活的架构框架,旨在学习可根据可用资源动态调整计算开销的高效视频编码器。其核心是受马特约什卡表示学习启发的自适应变压器块,可在推理时调整隐藏嵌入维度。我们展示,基于大规模Ego4D数据集中的视频叙述对训练的AdaVid-EgoVLP,在短视频-语言基准测试中仅使用一半算力即可匹配标准EgoVLP的性能,并在相同算力下表现更优。我们进一步在具有挑战性的Diving48分类基准上探索了帧数与计算量之间的权衡,表明AdaVid允许在不超出计算限制的情况下使用更多帧。为处理更长视频,我们还提出一种轻量级分层网络,聚合短片段特征,在多个长视频基准上实现了计算效率与准确率的良好平衡。

原文摘要 · Abstract (English)

Contrastive video-language pretraining has demonstrated great success in learning rich and robust video representations. However, deploying such video encoders on compute-constrained edge devices remains challenging due to their high computational demands. Additionally, existing models are typically trained to process only short video clips, often limited to 4 to 64 frames. In this paper, we introduce AdaVid, a flexible architectural framework designed to learn efficient video encoders that can dynamically adapt their computational footprint based on available resources. At the heart of AdaVid is an adaptive transformer block, inspired by Matryoshka Representation Learning, which allows the model to adjust its hidden embedding dimension at inference time. We show that AdaVid-EgoVLP, trained on video-narration pairs from the large-scale Ego4D dataset, matches the performance of the standard EgoVLP on short video-language benchmarks using only half the compute, and even outperforms EgoVLP when given equal computational resources. We further explore the trade-off between frame count and compute on the challenging Diving48 classification benchmark, showing that AdaVid enables the use of more frames without exceeding computational limits. To handle longer videos, we also propose a lightweight hierarchical network that aggregates short clip features, achieving a strong balance between compute efficiency and accuracy across several long video benchmarks.

视频预训练自适应计算边缘部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。