首个统一音视频文本的通用嵌入模型，支持任意模态间互搜与指令感知检索。
WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM
- 分层融合特征+多任务联合训练，构建统一跨模态空间。
- 在MMEB-v2上刷新视频基准，音视频到音频检索性能领先。
- 支持指令驱动嵌入，适合需要灵活交互的多模态应用。
尽管多模态大语言模型的嵌入具备通用表示能力，但其在音频、视频等动态模态上的应用仍不充分。本文提出WAVE（统一且通用的音视频嵌入），首个基于大语言模型的统一文本、音频、视频表示框架。WAVE采用新型分层特征融合策略与联合多模态多任务训练，实现两大核心能力：任意模态间的跨模态检索，以及根据用户指令生成的提示感知嵌入。实验表明，WAVE在MMEB-v2视频基准上达到新SOTA，音视频到音频检索表现优异。其提示感知特性在多模态问答中显著优于现有嵌入模型。消融实验验证了联合训练的有效性，所有模态性能均提升。本文还引入全新音视频通用学习基准，为任意模态间应用开辟广阔可能。代码与模型已开源。
原文摘要 · Abstract (English)
While embeddings from multimodal large language models (LLMs) excel as general-purpose representations, their application to dynamic modalities like audio and video remains underexplored. We introduce WAVE (\textbf{u}nified \& \textbf{v}ersatile \textbf{a}udio-\textbf{v}isual \textbf{e}mbeddings), the first LLM-based embedding that creates a unified representation space for text, audio, and video modalities. WAVE employs a novel hierarchical feature fusion strategy and a joint multi-modal, multi-task training approach to enable two key capabilities: any-to-any cross-modal retrieval and the generation of prompt-aware embeddings tailored to user instructions. Experimentally, WAVE sets a new state-of-the-art on the MMEB-v2 video benchmark and achieves superior results in audio and video-to-audio retrieval. Its prompt-aware nature also yields remarkable performance in multimodal question answering, significantly outperforming existing embedding models. Ablation studies validate our joint training strategy, demonstrating improved performance across all modalities. With a newly introduced benchmark for versatile audio-visual learning, WAVE opens up broad possibilities for cross-modal, any-to-any applications. Our code and checkpoints are released at \href{https://github.com/TCL606/WAVE}{https://github.com/TCL606/WAVE}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。