提出新型音频编码方法NDVQ，提升低带宽下的音质与语音合成效果。
NDVQ: Robust Neural Audio Codec with Normal Distribution-Based Vector Quantization
- 用正态分布表示码本，通过学习方差显式增加码字间距。
- 在极低带宽下，音质和零样本语音合成性能优于EnCodec等基线。
- 适合对鲁棒性要求高的音频压缩与语音生成任务。
基于向量量化（VQ）的离散音频编码模型在音频压缩和自回归音频生成中取得了显著进展。然而，现有模型在感知质量与信号失真方面仍面临挑战，尤其是在极低带宽下，其根源在于VQ码本对噪声敏感。这一退化问题严重影响了编码器驱动的语音合成等下游任务。为解决此问题，本文提出一种新型向量量化方法——基于正态分布的向量量化（NDVQ），通过学习方差显式引入码字间的距离边界。具体而言，将波形映射到潜在空间，并通过选择最可能的正态分布进行量化，每个码本条目代表一个由均值和方差定义的独特正态分布。利用这些基于分布的量化码本，解码器可重建输入波形。NDVQ在训练中结合了额外的分布相关损失、重建损失与判别损失。实验表明，在极低带宽场景下，NDVQ在音频质量和零样本语音合成（zero-shot TTS）方面均优于现有音频压缩基线，如EnCodec。
原文摘要 · Abstract (English)
Built upon vector quantization (VQ), discrete audio codec models have achieved great success in audio compression and auto-regressive audio generation. However, existing models face substantial challenges in perceptual quality and signal distortion, especially when operating in extremely low bandwidth, rooted in the sensitivity of the VQ codebook to noise. This degradation poses significant challenges for several downstream tasks, such as codec-based speech synthesis. To address this issue, we propose a novel VQ method, Normal Distribution-based Vector Quantization (NDVQ), by introducing an explicit margin between the VQ codes via learning a variance. Specifically, our approach involves mapping the waveform to a latent space and quantizing it by selecting the most likely normal distribution, with each codebook entry representing a unique normal distribution defined by its mean and variance. Using these distribution-based VQ codec codes, a decoder reconstructs the input waveform. NDVQ is trained with additional distribution-related losses, alongside reconstruction and discrimination losses. Experiments demonstrate that NDVQ outperforms existing audio compression baselines, such as EnCodec, in terms of audio quality and zero-shot TTS, particularly in very low bandwidth scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。