用统一模型同时解决语音增强、回声消除等多类语音前端任务。
UniFlow: Unifying Speech Front-End Tasks via Continuous Generative Modeling
- 基于连续生成建模,在共享隐空间中统一处理多种语音任务。
- 在多个公开数据集上优于现有最佳方法,性能提升稳定。
- 适合需要整合多种语音处理功能的系统开发者使用。
生成建模在图像、视频和音频领域取得显著成功,展现出强大的统一表征学习能力。然而,语音前端任务如语音增强(SE)、目标说话人提取(TSE)、声学回声消除(AEC)和语言查询源分离(LASS)仍主要依赖独立的任务专用方案,导致工程重复、性能不一致且难以扩展。为此,我们提出UniFlow,一个采用连续生成建模的统一框架,可在共享隐空间中处理多样化的语音前端任务。具体地,UniFlow利用波形变分自编码器(VAE)学习原始音频的紧凑隐表示,并结合扩散变压器(DiT)预测隐状态更新。通过可学习的条件嵌入(以任务ID索引),在最大化参数共享的同时保留任务特异性。为平衡性能与效率,我们在隐空间内比较了三种生成目标:去噪扩散、流匹配和均值流。在多个公开基准测试中,UniFlow持续优于现有最优基线。其统一的隐表示形式与条件设计使其易于扩展至新任务,为构建和扩展生成式语音处理流水线提供集成基础。为促进后续研究,代码将开源。
原文摘要 · Abstract (English)
Generative modeling has recently achieved remarkable success across image, video, and audio domains, demonstrating powerful capabilities for unified representation learning. Yet speech front-end tasks such as speech enhancement (SE), target speaker extraction (TSE), acoustic echo cancellation (AEC), and language-queried source separation (LASS) remain largely tackled by disparate, task-specific solutions. This fragmentation leads to redundant engineering effort, inconsistent performance, and limited extensibility. To address this gap, we introduce UniFlow, a unified framework that employs continuous generative modeling to tackle diverse speech front-end tasks in a shared latent space. Specifically, UniFlow utilizes a waveform variational autoencoder (VAE) to learn a compact latent representation of raw audio, coupled with a Diffusion Transformer (DiT) that predicts latent updates. To differentiate the speech processing task during the training, learnable condition embeddings indexed by a task ID are employed to enable maximal parameter sharing while preserving task-specific adaptability. To balance model performance and computational efficiency, we investigate and compare three generative objectives: denoising diffusion, flow matching, and mean flow within the latent domain. We validate UniFlow on multiple public benchmarks, demonstrating consistent gains over state-of-the-art baselines. UniFlow's unified latent formulation and conditional design make it readily extensible to new tasks, providing an integrated foundation for building and scaling generative speech processing pipelines. To foster future research, we will open-source our codebase.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。