用生成模型实现文本查询的声音分离,效果更好且更快。
FlowSep: Language-Queried Sound Separation with Rectified Flow Matching
- 基于修正流匹配的生成方法,从噪声中逐步生成目标声音特征。
- 在1680小时数据上训练,优于当前最佳模型,分离更完整无缺陷。
- 适合需要高质量音频分离与快速推理的应用场景。
语言查询音频源分离(LASS)旨在通过文本描述分离目标声音。现有方法多采用判别式模型,如时频掩码,但面对重叠音轨时易产生频谱空洞或分离不全等伪影。修正流匹配(RFM)是一种生成模型,具备优良理论性质和简洁性,尚未用于声音分离。本文提出FlowSep,基于RFM的生成模型,学习变分自编码器(VAE)潜空间中从噪声到目标源特征的线性流轨迹。推理时,通过预训练的VAE解码器将生成的潜变量重建为梅尔频谱图,并由预训练声码器合成波形。模型在1,680小时音频数据上训练,经主观与客观评估,在多个基准上超越当前最优模型。结果还显示,FlowSep在分离质量与推理效率上均优于基于扩散模型的LASS方法,展现出强大潜力。代码、预训练模型与演示见:https://audio-agi.github.io/FlowSep_demo/
原文摘要 · Abstract (English)
Language-queried audio source separation (LASS) focuses on separating sounds using textual descriptions of the desired sources. Current methods mainly use discriminative approaches, such as time-frequency masking, to separate target sounds and minimize interference from other sources. However, these models face challenges when separating overlapping soundtracks, which may lead to artifacts such as spectral holes or incomplete separation. Rectified flow matching (RFM), a generative model that establishes linear relations between the distribution of data and noise, offers superior theoretical properties and simplicity, but has not yet been explored in sound separation. In this work, we introduce FlowSep, a new generative model based on RFM for LASS tasks. FlowSep learns linear flow trajectories from noise to target source features within the variational autoencoder (VAE) latent space. During inference, the RFM-generated latent features are reconstructed into a mel-spectrogram via the pre-trained VAE decoder, followed by a pre-trained vocoder to synthesize the waveform. Trained on 1,680 hours of audio data, FlowSep outperforms the state-of-the-art models across multiple benchmarks, as evaluated with subjective and objective metrics. Additionally, our results show that FlowSep surpasses a diffusion-based LASS model in both separation quality and inference efficiency, highlighting its strong potential for audio source separation tasks. Code, pre-trained models and demos can be found at: https://audio-agi.github.io/FlowSep_demo/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。