将一致性轨迹模型引入语音增强,实现快速高质语音修复。
Schrödinger Bridge Consistency Trajectory Models for Speech Enhancement
- 结合薛定谔桥与一致性轨迹模型,提升语音增强推理速度。
- 实时因子(RTF)提升约16倍,且质量与速度平衡良好。
- 适合需要低延迟语音处理的场景,如实时通话、语音助手。
基于扩散模型的语音增强技术能有效提升噪声环境下语音质量。近期,薛定谔桥(Schrödinger Bridge, SB)被用于解决前向过程终点与反向过程起点不匹配的问题,从而改善语音质量。然而,该方法推理速度慢,需大量函数求值(NFE)才能获得高质量结果。一致性模型(CMs)虽通过一致性训练加速推理,但增加步数时生成质量并未提升。为此,一致性轨迹模型(CTMs)不仅加快推理速度,还能在质量与速度间保持良好权衡。SoundCTM已证明其在声学生成中的可行性。本文提出薛定谔桥一致性轨迹模型(SBCTM),将CTM技术应用于语音增强中的薛定谔桥。同时,在原有训练框架中引入含感知损失的新型辅助损失。实验表明,相比传统薛定谔桥,SBCTM实现约16倍的实时因子(RTF)提升。其优良的质量-速度权衡使得多步精炼仅在单步推理不足时启用,显著提高效率。代码、预训练模型及音频样例见:https://github.com/sony/sbctm/
原文摘要 · Abstract (English)
Speech enhancement (SE) utilizing diffusion models is a promising technology that improves speech quality in noisy speech data. Furthermore, the Schrödinger bridge (SB) has recently been used in diffusion-based SE to improve speech quality by resolving a mismatch between the endpoint of the forward process and the starting point of the reverse process. However, the SB still exhibits slow inference owing to the necessity of a large number of function evaluations (NFE) for inference to obtain high-quality results. While Consistency Models (CMs) address this issue by employing consistency training that uses distillation from pretrained models in the field of image generation, it does not improve generation quality when the number of steps increases. As a solution to this problem, Consistency Trajectory Models (CTMs) not only accelerate inference speed but also maintain a favorable trade-off between quality and speed. Furthermore, SoundCTM demonstrates the applicability of CTM techniques to the field of sound generation. In this paper, we present Schrödinger bridge Consistency Trajectory Models (SBCTM) by applying the CTM's technique to the Schrödinger bridge for SE. Additionally, we introduce a novel auxiliary loss, including a perceptual loss, into the original CTM's training framework. As a result, SBCTM achieves an approximately 16x improvement in the real-time factor (RTF) compared to the conventional Schrödinger bridge for SE. Furthermore, the favorable trade-off between quality and speed in SBCTM allows for time-efficient inference by limiting multi-step refinement to cases where 1-step inference is insufficient. Our code, pretrained models, and audio samples are available at https://github.com/sony/sbctm/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。