用大模型生成跨模态音频嵌入，提升长文本检索效果
Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval
- 用提示词和上下文学习让大模型生成通用多模态嵌入
- 在文本-音频检索中超越传统CLAP模型，尤其擅长长文本
- 适合需要强语言理解的多模态检索任务
近年来多模态大语言模型（MLLMs）取得显著进展，但其在声学领域的多模态信息表征能力仍待探索。本文提出Vela框架，通过精心设计的提示词与选定的上下文学习示例，使MLLM能有效生成通用多模态嵌入，弥合不同模态间的差距。我们采用单模态训练策略，仅使用文本对进行训练。实验表明，Vela在标准文本-音频检索任务中优于传统CLAP模型。此外，我们构建了新基准，揭示了CLAP在处理长文本与复杂检索任务中的局限性。相比之下，Vela凭借MLLM的强大能力，在这些场景下表现出稳健性能。代码即将开源。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have seen substantial progress in recent years. However, their ability to represent multimodal information in the acoustic domain remains underexplored. In this work, we introduce Vela, a novel framework designed to adapt MLLMs for the generation of universal multimodal embeddings. By leveraging MLLMs with specially crafted prompts and selected in-context learning examples, Vela effectively bridges the modality gap across various modalities. We then propose a single-modality training approach, where the model is trained exclusively on text pairs. Our experiments show that Vela outperforms traditional CLAP models in standard text-audio retrieval tasks. Furthermore, we introduce new benchmarks that expose CLAP models' limitations in handling long texts and complex retrieval tasks. In contrast, Vela, by harnessing the capabilities of MLLMs, demonstrates robust performance in these scenarios. Our code will soon be available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。