用音频CLIP增强关键词提示，让零样本音频描述更准确
MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Models
- 用音频CLIP提取特征生成结构化提示，引导LLM生成描述
- 使用MAGIC搜索使自然语言生成得分提升35%（4.7→7.3）
- 单个关键词提示效果最佳，无关键词时性能下降50%
自动音频描述（AAC）为音频片段生成文本描述，但因数据集规模远小于图像描述任务而面临挑战。为此，我们提出一种零样本AAC系统，利用预训练模型实现无需大量训练。该方法通过预训练音频CLIP模型提取听觉特征，并生成结构化提示，引导大语言模型（LLM）进行描述生成。与传统贪婪解码不同，本方法通过音频CLIP模型优化词元选择，确保生成内容与音频内容对齐。实验表明，在WavCaps模型上使用MAGIC搜索可使自然语言生成平均得分提升35%（从4.7升至7.3）。性能高度依赖音频-文本匹配模型与关键词选择，最优结果出现在使用单个关键词提示时，若不使用关键词列表，性能下降50%。
原文摘要 · Abstract (English)
Automated Audio Captioning (AAC) generates captions for audio clips but faces challenges due to limited datasets compared to image captioning. To overcome this, we propose the zero-shot AAC system that leverages pre-trained models, eliminating the need for extensive training. Our approach uses a pre-trained audio CLIP model to extract auditory features and generate a structured prompt, which guides a Large Language Model (LLM) in caption generation. Unlike traditional greedy decoding, our method refines token selection through the audio CLIP model, ensuring alignment with the audio content. Experimental results demonstrate a 35% improvement in NLG mean score (from 4.7 to 7.3) using MAGIC search with the WavCaps model. The performance is heavily influenced by the audio-text matching model and keyword selection, with optimal results achieved using a single keyword prompt, and a 50% performance drop when no keyword list is used.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。