用自监督表征重建提升语音编解码的可懂性和实时性
Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec
- 通过重建自监督提取的语音表征来优化训练
- 30万步即达优秀效果，零前瞻延迟实现实时传输
- 适合追求高可懂度与低延迟的语音编码研究者
针对基于梅尔频谱图重建的神经音频编解码器在语音可懂性方面表现不佳的问题，本文提出自监督表征重建（SSRR）损失。该方法显著加速收敛，在单张H200 GPU上仅需30万次训练步骤即可达到优异性能。通过从编解码输出中重建经语义蒸馏的自监督表征，有效提升了语音可懂性。更重要的是，SSRR使基于Transformer的流式编解码器无需额外前瞻（zero-lookahead），实现真正实时部署。在LibriSpeech test-clean数据集上，JHCodec在保持零前瞻和低端到端延迟的前提下，取得了最优的字错误率（WER）与字符错误率（CER）。项目代码、训练流程及演示已开源至GitHub。
原文摘要 · Abstract (English)
Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation improves encoded representations, it does not guarantee content preservation in reconstructed speech. In this work, we demonstrate that self-supervised representation reconstruction (SSRR) loss fundamentally improves codec training and performance. First, SSRR significantly accelerates convergence, enabling competitive results after 300k training steps on a single H200 GPU. Second, it enhances intelligibility by reconstructing distilled self-supervised representations from codec outputs. Third, SSRR enables high intelligibility without additional lookahead in streaming Transformer-based codecs, allowing a zero-lookahead architecture for real-time deployment. On LibriSpeech test-clean, JHCodec achieves the best WER and CER among the evaluated codecs while maintaining zero lookahead and low end-to-end latency. We open-source the full implementation, training pipeline, and demo on GitHubh ttps://github.com/jhcodec843/jhcodec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。