arXiv:2510.18744eess.AScs.LG2025-10被引 3

用扩散模型实现低延迟在线语音增强,仅需一次调用即可处理音频流。

Diffusion Buffer for Online Generative Speech Enhancement

  • 将物理时间与扩散步数对齐,逐帧逐步去噪,实现在线处理。
  • 算法延迟从320-960毫秒降至32-176毫秒,性能反而提升。
  • 支持灵活调节延迟与音质的权衡,适合消费级设备实时应用。

在线语音增强传统上依赖预测模型,其优势在于每帧输入仅需一次模型调用。相比之下,生成式语音增强模型通常需要多次调用,计算开销过大,难以满足在线场景需求。本文提出扩散缓冲器(Diffusion Buffer),一种基于扩散的生成式语音增强模型,仅需对每帧输入进行一次神经网络调用,即可在消费级GPU上实现在线处理。核心思想是将物理时间与扩散时间步对齐,通过物理时间逐步去噪,历史帧去噪程度更高。因此,输出帧存在由扩散缓冲器决定的延迟,并具有对应预览期。本文扩展此前工作,设计了一种2D卷积UNet架构,专门适配扩散缓冲器的预览机制。实验表明,该结构在低算法延迟下显著提升性能。此外,使用数据预测损失替代去噪得分匹配损失,可在推理阶段灵活调控算法延迟与音质之间的权衡。改进后的扩散缓冲器将算法延迟从320–960毫秒大幅降低至32–176毫秒,同时性能不降反升。此前已证明离线生成式扩散模型在未见噪声语音上优于预测方法,本研究进一步验证在线版扩散缓冲器同样在未见数据上超越其预测对应模型。

原文摘要 · Abstract (English)

Online Speech Enhancement was mainly reserved for predictive models. A key advantage of these models is that for an incoming signal frame from a stream of data, the model is called only once for enhancement. In contrast, generative Speech Enhancement models often require multiple calls, resulting in a computational complexity that is too high for many online speech enhancement applications. This work presents the Diffusion Buffer, a generative diffusion-based Speech Enhancement model which only requires one neural network call per incoming signal frame from a stream of data and performs enhancement in an online fashion on a consumer-grade GPU. The key idea of the Diffusion Buffer is to align physical time with Diffusion time-steps. The approach progressively denoises frames through physical time, where past frames have more noise removed. Consequently, an enhanced frame is output to the listener with a delay defined by the Diffusion Buffer, and the output frame has a corresponding look-ahead. In this work, we extend upon our previous work by carefully designing a 2D convolutional UNet architecture that specifically aligns with the Diffusion Buffer's look-ahead. We observe that the proposed UNet improves performance, particularly when the algorithmic latency is low. Moreover, we show that using a Data Prediction loss instead of Denoising Score Matching loss enables flexible control over the trade-off between algorithmic latency and quality during inference. The extended Diffusion Buffer equipped with a novel NN and loss function drastically reduces the algorithmic latency from 320 - 960 ms to 32 - 176 ms with an even increased performance. While it has been shown before that offline generative diffusion models outperform predictive approaches in unseen noisy speech data, we confirm that the online Diffusion Buffer also outperforms its predictive counterpart on unseen noisy speech data.

语音增强扩散模型在线处理低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。