构建23项视频嵌入评测基准，揭示多模态模型优劣与音频作用条件。
MVEB: Massive Video Embedding Benchmark
- 设计跨任务视频嵌入评测框架，涵盖分类、检索等23项任务。
- 发现音频提升效果依赖标注来源：双模态标注时增益6分，单视觉标注时反而下降。
- 适配MTEB生态，支持文本图像音视频统一评估，适合多模态研究者使用。
我们提出大规模视频嵌入基准（MVEB），包含23项任务，覆盖分类、零样本分类、聚类、成对分类、检索及视频问答。评估33个模型后发现：基于多模态大模型（MLLM）的嵌入在分类、聚类、成对分类和问答任务上表现最优；多模态对齐方法在检索和零样本分类中占优；未经过对比学习优化的生成式MLLM在跨模态任务中严重退化。对比仅视频与音视频联合输入的实验表明，音频贡献取决于数据标注来源：当标签来自双模态时音频提升性能，而仅基于视觉标注的数据中音频会带来负面影响，该差距在各模型族中均稳定存在约6分。MVEB源自184项任务的MVEB+池，兼顾任务多样性并降低评估成本。它已集成至MTEB生态系统，实现文本、图像、音频、视频的统一评估。我们已开源MVEB及全部184项任务的代码、评测工具与排行榜，地址为https://github.com/embeddings-benchmark/mteb。
原文摘要 · Abstract (English)
We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric question answering. We evaluate 33 models and find that no single model dominates: MLLM-based embeddings lead on classification, clustering, pair classification, and QA; multimodal binding leads on retrieval and zero-shot classification; generative MLLMs without contrastive adaptation collapse on cross-modal tasks. Paired video-only vs. audio+video evaluations show that audio's contribution depends on dataset annotation provenance: audio helps when labels were produced from both modalities and hurts when they were produced from visuals alone, a six-point gap consistent across model families. MVEB is derived from MVEB+, a 184-task pool, and is designed to maintain task diversity while reducing evaluation cost. It integrates into the MTEB ecosystem for unified evaluation across text, image, audio, and video. We release MVEB and all 184 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。