用语义代码本提升语音压缩效率，让模型更懂语言
Semantic Codebooks as Effective Priors for Neural Speech Compression
- 首层量化器从HuBERT提取语义令牌，后续层处理声学残差
- 0.95kbps下比DAC更低码率，识别错误率降低23%
- 适合需要低码率高识别率的语音任务
传统语音编码器以波形保真度为目标，过度分配比特保留可由语言结构推断的声学细节，导致压缩效率低下且下游识别性能不佳。本文提出SemDAC，一种基于语义代码本的神经语音编码器。其在残差向量量化（RVQ）堆栈中，第一层量化器通过蒸馏HuBERT特征生成捕捉音位内容的语义令牌，后续量化器建模声学残差。解码器采用FiLM条件控制，根据语义令牌重建音频，提升了声学代码本的使用效率。尽管设计简单，该方法显著优于DAC：在感知指标上表现更优，且在使用Whisper进行语音识别时获得更低的词错误率（WER），同时运行于更低码率（如0.95 kbps vs. DAC的2.5 kbps）。结果表明，语义代码本为神经语音压缩提供了有效归纳偏置，生成紧凑且利于识别的表示。
原文摘要 · Abstract (English)
Speech codecs are traditionally optimized for waveform fidelity, allocating bits to preserve acoustic detail even when much of it can be inferred from linguistic structure. This leads to inefficient compression and suboptimal performance on downstream recognition tasks. We propose SemDAC, a semantic-aware neural audio codec that leverages semantic codebooks as effective priors for speech compression. In SemDAC, the first quantizer in a residual vector quantization (RVQ) stack is distilled from HuBERT features to produce semantic tokens that capture phonetic content, while subsequent quantizers model residual acoustics. A FiLM-conditioned decoder reconstructs audio conditioned on the semantic tokens, improving efficiency in the use of acoustic codebooks. Despite its simplicity, this design proves highly effective: SemDAC outperforms DAC across perceptual metrics and achieves lower WER when running Whisper on reconstructed speech, all while operating at substantially lower bitrates (e.g., 0.95 kbps vs. 2.5 kbps for DAC). These results demonstrate that semantic codebooks provide an effective inductive bias for neural speech compression, producing compact yet recognition-friendly representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。