arXiv:2409.13152eess.AScs.SD2024-09被引 7

用无字幕音频数据训练语音提取模型,提升文本查询的音效分离效果。

Leveraging Audio-Only Data for Text-Queried Target Sound Extraction

  • 用预训练模型将无字幕音频转为嵌入向量,替代文本标注进行训练。
  • 加入嵌入层随机丢弃,避免模型过拟合音频特征。
  • 实验证明该方法效果接近有标注数据,适合数据稀缺场景。

文本查询的目标声音提取(TSE)旨在从混合音频中提取由自然语言描述指定的声音源。虽然大规模的图文配对数据更理想,但高质量的图文对数量有限,制约了数据规模扩展。为此,本文探索如何利用无任何字幕的纯音频数据来扩大训练数据量。一种直接方法是使用联合音视频嵌入模型(如CLAP)作为查询编码器,通过真实音频获得的嵌入向量训练TSE模型,推理时切换至文本编码器接收文本查询。尽管若音视频嵌入空间对齐良好此法可行,但实践中因嵌入包含领域特异性信息,导致模型过拟合音频查询。本文研究多种防过拟合方法,发现简单的嵌入丢弃(dropout)即可有效缓解该问题。大量实验表明,使用无字幕音频配合嵌入丢弃,训练效果与使用文字标注相当,可有效提升文本查询的TSE模型性能。

原文摘要 · Abstract (English)

The goal of text-queried target sound extraction (TSE) is to extract from a mixture a sound source specified with a natural-language caption. While it is preferable to have access to large-scale text-audio pairs to address a variety of text prompts, the limited number of available high-quality text-audio pairs hinders the data scaling. To this end, this work explores how to leverage audio-only data without any captions for the text-queried TSE task to potentially scale up the data amount. A straightforward way to do so is to use a joint audio-text embedding model, such as the contrastive language-audio pre-training (CLAP) model, as a query encoder and train a TSE model using audio embeddings obtained from the ground-truth audio. The TSE model can then accept text queries at inference time by switching to the text encoder. While this approach should work if the audio and text embedding spaces in CLAP were well aligned, in practice, the embeddings have domain-specific information that causes the TSE model to overfit to audio queries. We investigate several methods to avoid overfitting and show that simple embedding-manipulation methods such as dropout can effectively alleviate this issue. Extensive experiments demonstrate that using audio-only data with embedding dropout is as effective as using text captions during training, and audio-only data can be effectively leveraged to improve text-queried TSE models.

语音提取无监督训练CLAP音频理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。