低帧率语音编码器，兼顾音质与生成效率
DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation
- 双流编码融合自监督语义与波形信息
- 低帧率下仍保持高音质，优于Mimi等模型
- 适合需要高效语音生成的场景
神经音频编码器是基于语言模型的语音生成基础。传统方法在帧率与音质间存在权衡。本文提出DualCodec，一种低帧率、语义增强的编码模型。现有方法将富含语义的自监督学习（SSL）表征压缩至第一层编码令牌。本工作设计双流编码框架，将SSL与波形表示端到端融合。该结构提升第一层编码的语义信息，使编码系统在低帧率下仍维持高音质。低帧率编码有助于提升语音生成效率。在音频编码与语音生成任务上的实验表明，DualCodec优于当前先进模型，如Mimi Codec、SpeechTokenizer、DAC和Encodec。演示与代码已公开：https://dualcodec.github.io, https://github.com/jiaqili3/DualCodec
原文摘要 · Abstract (English)
Neural audio codecs form the foundational building blocks for language model (LM)-based speech generation. Typically, there is a trade-off between frame rate and audio quality. This study introduces a low-frame-rate, semantically enhanced codec model. Existing approaches distill semantically rich self-supervised (SSL) representations into the first-layer codec tokens. This work proposes DualCodec, a dual-stream encoding approach that integrates SSL and waveform representations within an end-to-end codec framework. In this setting, DualCodec enhances the semantic information in the first-layer codec and enables the codec system to maintain high audio quality while operating at a low frame rate. Note that a low-frame-rate codec improves the efficiency of speech generation. Experimental results on audio codec and speech generation tasks confirm the effectiveness of the proposed DualCodec compared to state-of-the-art codec systems, such as Mimi Codec, SpeechTokenizer, DAC, and Encodec. Demos are available at: https://dualcodec.github.io, code is available at: https://github.com/jiaqili3/DualCodec
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。