用大模型分析噪音类型，提升嘈杂环境下的声音事件检测效果
Leveraging LLM and Text-Queried Separation for Noise-Robust Sound Event Detection
- 用大模型识别噪声类型，生成增强数据用于鲁棒训练
- 在混响噪声下提升检测准确率，尤其在重叠事件场景表现更好
- 适合研究噪声鲁棒性或跨模态音频处理的开发者
声音事件检测（SED）在嘈杂环境中面临挑战，因重叠声音会掩盖目标事件。语言查询音频分离（LASS）旨在从噪声片段中分离出目标声音事件，但当目标声音未知时，该方法在噪声测试集上性能下降。为此，本文利用大语言模型（LLMs）分析和总结声学数据，通过识别特定噪声类型实现噪声增强，用于噪声鲁棒性微调。微调后的模型将片段级事件预测转为文本查询，输入LASS模型进行分离。实验表明，该方法显著提升了嘈杂环境中的SED性能。本工作是大模型应用于噪声鲁棒SED的早期探索，为解决重叠事件问题提供了新方向。代码与预训练模型已公开于 https://github.com/apple-yinhan/Noise-robust-SED。
原文摘要 · Abstract (English)
Sound Event Detection (SED) is challenging in noisy environments where overlapping sounds obscure target events. Language-queried audio source separation (LASS) aims to isolate the target sound events from a noisy clip. However, this approach can fail when the exact target sound is unknown, particularly in noisy test sets, leading to reduced performance. To address this issue, we leverage the capabilities of large language models (LLMs) to analyze and summarize acoustic data. By using LLMs to identify and select specific noise types, we implement a noise augmentation method for noise-robust fine-tuning. The fine-tuned model is applied to predict clip-wise event predictions as text queries for the LASS model. Our studies demonstrate that the proposed method improves SED performance in noisy environments. This work represents an early application of LLMs in noise-robust SED and suggests a promising direction for handling overlapping events in SED. Codes and pretrained models are available at https://github.com/apple-yinhan/Noise-robust-SED.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。