用神经音频编解码器的隐空间提升语音增强效果
Modeling strategies for speech enhancement in the latent space of a neural audio codec
- 在编解码器隐空间中直接预测连续向量，优于离散标记
- 非自回归模型兼顾音质与效率，更适合实际应用
- 微调编码器能显著提升增强效果，但会降低重建质量
神经音频编解码器（NAC）以连续向量或离散标记序列的形式提供紧凑的语音隐表示。本文研究了这两种表示作为监督语音增强训练目标时的性能对比。实验基于Conformer架构，考察了自回归与非自回归语音增强模型，以及仅微调NAC编码器的基线方法。结果表明：预测连续隐表示始终优于离散标记预测；自回归模型虽音质更高，但牺牲了可懂度和效率，非自回归模型更具实用性；加入编码器微调可获得最强的增强指标，但会降低编解码器重建质量。代码与音频样例已公开。
原文摘要 · Abstract (English)
Neural audio codecs (NACs) provide compact latent speech representations in the form of sequences of continuous vectors or discrete tokens. In this work, we investigate how these two types of speech representations compare when used as training targets for supervised speech enhancement. We consider both autoregressive and non-autoregressive speech enhancement models based on the Conformer architecture, as well as a simple baseline where the NAC encoder is simply fine-tuned for speech enhancement. Our experiments reveal three key findings: predicting continuous latent representations consistently outperforms discrete token prediction; autoregressive models achieve higher quality but at the expense of intelligibility and efficiency, making non-autoregressive models more attractive in practice; and adding encoder fine-tuning yields the strongest enhancement metrics overall, though at the cost of degraded codec reconstruction. The code and audio samples are available online.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。