将语音分解为语义、音色、语调和残差四路，低码率下实现高保真语音重建。
MSR-Codec: A Low-Bitrate Multi-Stream Residual Codec for High-Fidelity Speech Generation with Information Disentanglement
- 分四路编码：语义、音色、语调、残差，实现信息解耦
- 在低码率下重建语音质量达当前最优，且支持独立调控音色与语调
- 适合语音合成、语音转换等需灵活控制声学属性的任务
音频编解码器是现代语音生成系统的关键组件。本文提出一种低码率、多尺度残差编解码器，将语音编码为语义、音色、语调和残差四个独立流，实现了在竞争性低码率下的高保真语音重建，并具备天然的信息解耦能力。我们基于该编解码器构建了一个两阶段文本到语音（TTS）合成模型，尽管设计轻量且数据需求少，仍达到领先水平的词错误率（WER）和优于多个更大模型的说话人相似度。此外，该编解码器在语音转换任务中表现优异，可独立操控说话人音色与语调。相关推理代码、预训练模型及音频样例已开源至 https://github.com/herbertLJY/MSRCodec。
原文摘要 · Abstract (English)
Audio codecs are a critical component of modern speech generation systems. This paper introduces a low-bitrate, multi-scale residual codec that encodes speech into four distinct streams: semantic, timbre, prosody, and residual. This architecture achieves high-fidelity speech reconstruction at competitive low bitrates while demonstrating an inherent ability for information disentanglement. We construct a two-stage language model for text-to-speech (TTS) synthesis using this codec, which, despite its lightweight design and minimal data requirements, achieves a state-of-the-art Word Error Rate (WER) and superior speaker similarity compared to several larger models. Furthermore, the codec's design proves highly effective for voice conversion, enabling independent manipulation of speaker timbre and prosody. Our inference code, pre-trained models, and audio samples are available at https://github.com/herbertLJY/MSRCodec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。