用生成模型提升音频文本检索的泛化能力
DiffATR: Diffusion-based Generative Modeling for Audio-Text Retrieval
- 将音频文本检索建模为联合分布生成过程
- 在AudioCaps和Clotho上超越现有方法，尤其在域外场景表现优异
- 结合生成与对比学习，兼具强泛化与高精度
现有音频-文本检索（ATR）方法多为判别式模型，旨在最大化条件概率 p(候选|查询)，但忽略了查询的先验分布 p(查询)，导致难以识别分布外数据。本文从生成视角出发，建模音频与文本的联合概率 p(候选, 查询)。提出基于扩散模型的ATR框架DiffATR，通过逐步从噪声生成联合分布来实现检索。训练中同时优化生成损失与对比损失：生成器通过生成损失改进，特征提取器通过对比损失优化，融合两类方法优势。在AudioCaps和Clotho数据集上的实验表明，本方法性能优越；尤其值得注意的是，无需任何修改，DiffATR在域外检索场景中始终表现强劲。
原文摘要 · Abstract (English)
Existing audio-text retrieval (ATR) methods are essentially discriminative models that aim to maximize the conditional likelihood, represented as p(candidates|query). Nevertheless, this methodology fails to consider the intrinsic data distribution p(query), leading to difficulties in discerning out-of-distribution data. In this work, we attempt to tackle this constraint through a generative perspective and model the relationship between audio and text as their joint probability p(candidates,query). To this end, we present a diffusion-based ATR framework (DiffATR), which models ATR as an iterative procedure that progressively generates joint distribution from noise. Throughout its training phase, DiffATR is optimized from both generative and discriminative viewpoints: the generator is refined through a generation loss, while the feature extractor benefits from a contrastive loss, thus combining the merits of both methodologies. Experiments on the AudioCaps and Clotho datasets with superior performances, verify the effectiveness of our approach. Notably, without any alterations, our DiffATR consistently exhibits strong performance in out-of-domain retrieval settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。