用大模型统一处理音频文本检索,效果更好且省数据。
Scaling Audio-Text Retrieval with Multimodal Large Language Models
- 用多模态大模型做统一编码,解决小模型理解力不足问题。
- 仅用1%数据就超越顶尖模型,检索准确率显著提升。
- 适合需要高效音频文本匹配的开发者或研究者使用。
音频-文本检索对连接声音与自然语言至关重要。尽管对比双编码器架构(如CLAP)已有进展,但受限于小规模编码器容量,尤其在处理需推理或世界知识的复杂查询时表现不佳。本文提出AuroLA,一种新型对比式语言-音频预训练框架,将多模态大语言模型(MLLM)作为统一骨干网络。首先,构建可扩展数据流水线,从多个来源收集多样化音频,并通过自动化标注生成多粒度描述,涵盖长文本描述到结构化标签;其次,通过提示MLLM总结音频/文本输入,并以特殊标记的隐藏状态作为嵌入表示,设计新颖的混合NCE损失,结合多粒度监督与难负样本重加权,实现音频与多样化文本监督的稳健对齐;最后,设计基于MLLM的双向重排序模块,通过深度跨模态交互优化候选结果。大量实验表明,AuroLA持续优于当前最先进模型(包括近期提出的PE-AV),且仅需约PE-AV 1%的训练数据。此外,我们观察到数据量与模型容量的清晰缩放趋势,验证了MLLM作为统一骨干在音频-文本检索中的有效性。代码已开源:https://github.com/Jazzcharles/AuroLA。
原文摘要 · Abstract (English)
Audio-text retrieval is crucial for bridging acoustic signals and natural language. While contrastive dual-encoder architectures like CLAP have shown promise, they are fundamentally limited by the capacity of small-scale encoders. Specifically, the text encoders struggle to understand complex queries that require reasoning or world knowledge. In this paper, we propose AuroLA, a novel contrastive language-audio pre-training framework that re-purposes Multimodal Large Language Models (MLLMs) as a unified backbone for retrieval. Specifically, we make three contributions: (i) we construct a scalable data pipeline that curates diverse audio from multiple sources and generates multi-granular captions, ranging from long descriptions to structured tags, via automated annotation; (ii) we adapt an MLLM for retrieval by prompting it to summarize the audio/text input and using the hidden state of a special token as audio/text embeddings. For model training, we devise a novel Hybrid-NCE loss, which employs multi-granular supervision and hard-negative reweighting to robustly align audio with diverse textual supervision; and (iii) we design an MLLM-based bidirectional re-ranking module that refines retrieval candidates through deep cross-modal interaction. Extensive experiments demonstrate that AuroLA consistently outperforms state-of-the-art models, including the recent PE-AV, while utilizing only approximately 1% of PE-AV's training data. Lastly, we observe clear scaling trends regarding dataset size and model capacity, validating the effectiveness of MLLM as a unified backbone for audio-text retrieval. Code is available at https://github.com/Jazzcharles/AuroLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。