用单量化器实现24kHz语音0.3kbps超低带宽高保真重建
HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
- 设计专用向量量化空间，支持每秒24个离散语音标记
- 采用异构编解码架构，实现0.3kbps下顶尖语音重建质量
- 适合语音压缩、低带宽通信及生成模型轻量化部署
离散语音分词是语音编码的核心组件。然而，在大规模语音到语音系统中，多量化器并行流的复杂性和高时间维度编码器的计算开销带来显著挑战。本文提出HH-Codec，一种神经语音编码器，可在24 kHz音频上以每秒24个标记的速率实现极端压缩，且仅依赖单量化器推理。其核心是为语音建模精心设计的向量量化空间，兼顾压缩效率与信息损失最小化。在此基础上，提出异构编码器-解码器架构（Audio-VQ-Mel-Audio），通过双监督和渐进式训练提升重建稳定性和保真度。HH-Codec在语音重建任务中达到当前最优性能，仅需0.3 kbps超低带宽。我们进一步评估了码本利用率与生成模型适配性，并通过大量消融实验验证了各模块必要性。代码已开源：https://github.com/opendilab/HH-Codec。
原文摘要 · Abstract (English)
Discrete speech tokenization is a fundamental component in speech codecs. However, in large-scale speech-to-speech systems, the complexity of parallel streams from multiple quantizers and the computational cost of high-time-dimensional codecs pose significant challenges. In this paper, we introduce HH-Codec, a neural codec that achieves extreme compression at 24 tokens per second for 24 kHz audio while relying on single-quantizer inference. Our approach involves a carefully designed Vector Quantization space for Spoken Language Modeling, optimizing compression efficiency while minimizing information loss. Building on this, we propose an asymmetric encoder-decoder architecture (Audio-VQ-Mel-Audio) that leverages dual supervision and progressive training to enhance reconstruction stability and fidelity. HH-Codec achieves state-of-the-art performance in speech reconstruction with an ultra-low bandwidth of 0.3 kbps. We further evaluate its effectiveness in codebook utilization and generative model adaptation, with extensive ablations validating the necessity of each module. HH-Codec is available at https://github.com/opendilab/HH-Codec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。