用大模型自动分离未知音源，无需预设类别
OpenSep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation
- 通过文本反转从音频生成描述，识别混合中的声源
- 少样本提示大模型提取声源属性，实现新场景分离
- 多层级训练提升音画对齐，适合复杂真实场景
真实场景下的音频分离面临挑战，因现有模型存在过分离、欠分离及依赖预设音源的问题。本文提出OpenSep框架，利用大语言模型实现自动化音频分离，无需人工干预且突破音源限制。OpenSep通过文本反转技术，使用现成音频描述模型从混合音频生成字幕，有效解析出其中包含的声音源。随后通过少量样本提示大模型提取每个解析出音源的详细属性，从而在未见混合中实现分离。此外，我们引入多层级混合-分离训练框架，通过同时分离单音源与混合音频增强模态对齐。大量实验表明，OpenSep在复杂混合中精准分离新出现、未见过且数量可变的音源方面优于当前最先进方法。代码已开源：https://github.com/tanvir-utexas/OpenSep.git
原文摘要 · Abstract (English)
Audio separation in real-world scenarios, where mixtures contain a variable number of sources, presents significant challenges due to limitations of existing models, such as over-separation, under-separation, and dependence on predefined training sources. We propose OpenSep, a novel framework that leverages large language models (LLMs) for automated audio separation, eliminating the need for manual intervention and overcoming source limitations. OpenSep uses textual inversion to generate captions from audio mixtures with off-the-shelf audio captioning models, effectively parsing the sound sources present. It then employs few-shot LLM prompting to extract detailed audio properties of each parsed source, facilitating separation in unseen mixtures. Additionally, we introduce a multi-level extension of the mix-and-separate training framework to enhance modality alignment by separating single source sounds and mixtures simultaneously. Extensive experiments demonstrate OpenSep's superiority in precisely separating new, unseen, and variable sources in challenging mixtures, outperforming SOTA baseline methods. Code is released at https://github.com/tanvir-utexas/OpenSep.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。