提出混合式音频编解码器，兼顾语义与音质，推理速度提升3倍。
HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec
- 双流架构融合语义与声学分支，从预训练模型蒸馏语义信息。
- 在同域测试中语义专精度提升，重建质量与现有方法相当。
- 无需推理时调用预训练模型，适合跨语言零样本场景。
随着多模态大语言模型的兴起，神经音频编解码器作为语音分词器越来越受欢迎。新型编解码架构通过语义与声学特征解耦实现性能提升。现有方法主要分为两类：一类将自监督学习（SSL）表征中的语义信息蒸馏到第一层矢量量化（RVQ-1）；另一类则保持语义与声学特征的独立流。本文提出 HybridCodec，一种统一架构，结合两种范式：采用独立的语义与声学分支，并将 SSL 表征蒸馏至语义流。该设计在不依赖推断时的 SSL 模型前提下，实现强解耦。在同域测试集上，语义流（RVQ-1）表现更优；全层重建（RVQ-all）性能具有竞争力。实验表明其在跨域及零样本跨语言设置下具备鲁棒性，相较现有双流模型推理速度提升3倍。
原文摘要 · Abstract (English)
The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models. New codec architectures with semantic and acoustic disentanglement have emerged. There are two main approaches to introduce semantic information into codec models: one distills semantic information from SSL representations into the first RVQ layer, while the other maintains separate streams for semantic and acoustic features. We propose HybridCodec, a unified architecture that combines both paradigms. It employs separate semantic and acoustic branches while distilling SSL representations into the semantic stream. This design ensures strong disentanglement without requiring an SSL model during inference. HybridCodec shows superior semantic specialization (RVQ-1) on in-domain test set and competitive reconstruction (RVQ-all). We demonstrate its robustness in out-of-domain and zero-shot cross-lingual settings, achieving a 3x speedup over existing dual-stream models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。