用大模型增强音频描述生成,提升多样性和准确性。
SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs
- 通过重述数据增强扩展数据集,缓解音频-文本对稀缺问题。
- 在Clotho V2和AudioCaps上达到当前最佳性能,超越主流模型。
- 可插拔式优化策略,适合需要高质量音频描述的应用场景。
自动音频描述(AAC)旨在为输入音频信号生成自然的文本描述。近年来,音频预训练模型与大语言模型(LLMs)的发展显著提升了音频理解与文本推理能力,推动了AAC的进步。本文提出SLAM-AAC,通过大模型实现重述数据增强与CLAP-Refine优化。采用自监督EAT模型提取细粒度音频表征,并通过轻量线性层与文本嵌入对齐。使用LoRA适配器高效微调生成用的LLM。借鉴机器翻译中的回译思想,在预训练阶段对Clotho数据集实施重述增强,有效扩充数据并提升描述多样性。推理时引入可插拔的CLAP-Refine策略,利用多个解码结果的音频-文本相似度(基于CLAP模型)选择最优描述,类似语音识别中的n-best重评分。实验表明,SLAM-AAC在Clotho V2和AudioCaps上均达到当前最佳表现,显著超越以往主流模型。
原文摘要 · Abstract (English)
Automated Audio Captioning (AAC) aims to generate natural textual descriptions for input audio signals. Recent progress in audio pre-trained models and large language models (LLMs) has significantly enhanced audio understanding and textual reasoning capabilities, making improvements in AAC possible. In this paper, we propose SLAM-AAC to further enhance AAC with paraphrasing augmentation and CLAP-Refine through LLMs. Our approach uses the self-supervised EAT model to extract fine-grained audio representations, which are then aligned with textual embeddings via lightweight linear layers. The caption generation LLM is efficiently fine-tuned using the LoRA adapter. Drawing inspiration from the back-translation method in machine translation, we implement paraphrasing augmentation to expand the Clotho dataset during pre-training. This strategy helps alleviate the limitation of scarce audio-text pairs and generates more diverse captions from a small set of audio clips. During inference, we introduce the plug-and-play CLAP-Refine strategy to fully exploit multiple decoding outputs, akin to the n-best rescoring strategy in speech recognition. Using the CLAP model for audio-text similarity calculation, we could select the textual descriptions generated by multiple searching beams that best match the input audio. Experimental results show that SLAM-AAC achieves state-of-the-art performance on Clotho V2 and AudioCaps, surpassing previous mainstream models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。