arXiv:2502.13433cs.SDeess.AS2025-02ICML被引 2

仅用文本训练,让大模型理解音频。

MATS: An Audio Language Model under Text-only Supervision

  • 用文本监督训练,将音频语言共享空间映射到大模型空间。
  • 在多个音频任务上表现接近依赖音视频对的模型。
  • 适合想低成本训练音频理解模型的研究者。

大型音频-语言模型(LALMs)基于强大的大语言模型(LLMs),展现出卓越的音频理解和推理能力。然而,训练LALMs需要大量音频-语言配对数据,成本高昂。本文提出MATS,一种仅依赖文本监督的多音频任务音频-语言多模态大模型。通过利用预训练的音频-语言对齐模型(如CLAP),我们设计了一种纯文本训练策略,将共享的音频-语言潜在空间投影至LLM潜在空间,使LLM在训练中无需音频数据即可获得音频理解能力。为缩小CLAP中音频与语言嵌入间的模态差距,我们提出强相关噪声文本与音频(Santa)机制,将音频嵌入映射至CLAP语言嵌入空间,同时保留关键音频信息。大量实验表明,尽管MATS仅使用文本数据训练,其性能仍可与近期依赖大规模音视频对训练的LALMs相媲美。代码已公开于https://github.com/wangwen-banban/MATS。

原文摘要 · Abstract (English)

Large audio-language models (LALMs), built upon powerful Large Language Models (LLMs), have exhibited remarkable audio comprehension and reasoning capabilities. However, the training of LALMs demands a large corpus of audio-language pairs, which requires substantial costs in both data collection and training resources. In this paper, we propose \textbf{MATS}, an audio-language multimodal LLM designed to handle \textbf{M}ultiple \textbf{A}udio task using solely \textbf{T}ext-only \textbf{S}upervision. By leveraging pre-trained audio-language alignment models such as CLAP, we develop a text-only training strategy that projects the shared audio-language latent space into LLM latent space, endowing the LLM with audio comprehension capabilities without relying on audio data during training. To further bridge the modality gap between audio and language embeddings within CLAP, we propose the \textbf{S}trongly-rel\textbf{a}ted \textbf{n}oisy \textbf{t}ext with \textbf{a}udio (\textbf{Santa}) mechanism. Santa maps audio embeddings into CLAP language embedding space while preserving essential information from the audio input. Extensive experiments demonstrate that MATS, despite being trained exclusively on text data, achieves competitive performance compared to recent LALMs trained on large-scale audio-language pairs. The code is publicly available in \href{https://github.com/wangwen-banban/MATS}{https://github.com/wangwen-banban/MATS}.

音频理解文本监督大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。