用扩散模型弥合音频与文本的表示鸿沟，提升语音描述生成效果。
Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gap
- 基于扩散模型将音频嵌入映射到文本分布空间，轻量设计仅三块残差MLP。
- 在AudioCaps上零样本与全监督任务分别提升52.5%和7.5%指标。
- 首次将扩散模态桥接用于自动语音描述，适合多模态大模型融合场景。
对比音频-语言预训练可获得强大的联合表征，但持续存在的音频-文本模态鸿沟限制了多模态编码器与大语言模型（LLM）的耦合效果。本文提出Diffusion-Link，一种基于扩散模型的模态桥接模块，可生成性地将音频嵌入映射至文本嵌入分布。该模块以冻结的多模态编码器输出嵌入为输入，采用仅含三块残差MLP的轻量网络实现。为评估其对编码器-LLM耦合的影响，我们在自动语音描述（AAC）任务上进行测试，据我们所知，这是首次将基于扩散的模态桥接应用于AAC。实验报告两项结果：(1) 模态鸿沟分析显示，在相似性与几何标准下，Diffusion-Link相较已有扩散方法减少鸿沟最显著，并促使音频嵌入集体向文本分布迁移；(2) 下游任务中，将Diffusion-Link接入相同多模态LLM基线，在AudioCaps上实现零样本与全监督下的最新性能，相对提升分别达52.5%和7.5%，且无需外部知识。结果表明，弥合模态鸿沟对多模态编码器与LLM的有效结合至关重要，而基于扩散的模态桥接为超越知识检索导向的设计提供了新方向。代码将在接受后公开于https://github.com/DevKiHyun/Diffusion-Link。
原文摘要 · Abstract (English)
Contrastive audio-language pretraining yields powerful joint representations, yet a persistent audio-text modality gap limits the benefits of coupling multimodal encoders with large language models (LLMs). We present Diffusion-Link, a diffusion-based modality-bridging module that generatively maps audio embeddings into the text-embedding distribution. The module is trained at the output embedding from the frozen multimodal encoder and implemented as a lightweight network with three residual MLP blocks. To assess the effect of Diffusion-Link on multimodal encoder-LLM coupling, we evaluate on Automatic Audio Captioning (AAC); to our knowledge, this is the first application of diffusion-based modality bridging to AAC. We report two results. (1) Modality-gap analysis: on similarity and geometric criteria, Diffusion-Link reduces the modality gap the most among prior diffusion-based methods and shows a collective migration of audio embeddings toward the text distribution. (2) Downstream AAC: attaching Diffusion-Link to the same multimodal LLM baseline achieves state-of-the-art on AudioCaps in both zero-shot and fully supervised captioning without external knowledge, with relative gains up to 52.5% and 7.5%, respectively. These findings show that closing the modality gap is pivotal for effective coupling between multimodal encoders and LLMs, and diffusion-based modality bridging offers a promising direction beyond knowledge-retrieval-centric designs. Code will be released upon acceptance https://github.com/DevKiHyun/Diffusion-Link
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。