arXiv:2603.12837cs.SDcs.AI2026-03

融合掩码与流匹配,一步完成高质量目标说话人分离。

Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching

  • 两阶段设计:先用掩码处理缺失成分,再用单步流匹配重建语音。
  • 仅需8500万参数和一次推理,音质接近高阶扩散模型。
  • 适合低延迟语音分离场景,兼顾效率与听感质量。

目标说话人提取(TSE)从重叠语音中根据参考语句提取目标说话人的声音。现有基于掩码的方法轻量高效,但无法合成缺失内容,导致感知质量下降;而生成式TSE模型虽能通过扩散模型生成高质量语音,却需大量迭代步骤,计算开销大、延迟高。本文提出Mask2Flow-TSE,一种结合两者优势的两阶段框架。我们引入删除/插入(D/I)比例作为分析工具,发现早期流匹配步骤主要执行信号删除而非合成。基于此,将删除与插入解耦:第一阶段使用轻量卷积掩码模块处理主导的删除任务,第二阶段采用针对TSE改进的扩散变压器(DiT),以说话人条件进行单步插入重建。与以往从高斯噪声开始不同,本方法从掩码谱图出发,实现单步高质量重建。实验表明,该方法仅需8500万参数和一次推理,即可生成高质量语音,同时对原始单说话人输入影响极小。

原文摘要 · Abstract (English)

Target speaker extraction (TSE) extracts the target speaker's voice from overlapping speech given a reference utterance. Existing masking-based approaches are lightweight and effective but suffer from an inability to synthesize missing content, leading to degraded perceptual quality. On the other hand, recent generative TSE models typically synthesize high-quality speech with diffusion, but require numerous iterative steps resulting in high computational costs and latency. We propose Mask2Flow-TSE, a two-stage framework combining the strengths of both paradigms. We introduce the deletion/insertion (D/I) proportion, an analytical tool that reveals early flow steps predominantly remove signal components rather than synthesize them. Based on this finding, we decouple deletion from insertion: a masking-based module handles the deletion-dominant early steps, while a single flow-matching step performs the remaining insertion for high-quality reconstruction. Specifically, the first stage uses lightweight convolution for the masking module, while the second stage employs a Diffusion Transformer (DiT) adapted for TSE with speaker conditioning. Unlike prior approaches that start from Gaussian noise, our method starts from the masked spectrogram, enabling high-quality reconstruction in a single inference step. Experiments show that Mask2Flow-TSE produces high-quality extractions with only 85M parameters and one-step inference, while preserving clean single-speaker inputs with minimal degradation.

语音分离扩散模型低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。