arXiv:2510.00981cs.SD2025-10中稿 · ICLR被引 21

动态帧率神经音频编解码器,提升低帧率下语义信息保留能力。

FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates

  • 通过自适应合并相似语义帧,实现3-12.5Hz动态帧率
  • 在6.25Hz、8.3Hz和12.5Hz下保持高重建质量
  • 适用于语音语言模型与端到端语音合成

神经音频编解码器是语音语言模型的基础,需具备低帧率和语义与声学信息解耦的特性。更低的帧率可缩短序列长度,降低计算成本。尽管已有12.5Hz低帧率编解码器,但更低成本的帧率仍待探索。我们发现极低帧率下语义信息易丢失。为此提出FlexiCodec,采用动态帧率机制,在语义稀疏区域自适应合并相似帧以减少帧数。该架构结合ASR特征辅助的双流编码与Transformer瓶颈结构,支持推理时在3Hz至12.5Hz间调节帧率。在6.25Hz、8.3Hz和12.5Hz平均帧率下的实验表明,FlexiCodec在语义保留上优于基线系统,并保持高质量音频重建。其在基于语言模型的语音合成中亦表现有效。演示与代码见:https://flexicodec.github.io, https://github.com/amphionteam/flexicodec。

原文摘要 · Abstract (English)

Neural audio codecs are foundational to speech language models. It is expected to have a low frame rate and decoupled semantic and acoustic information. A lower frame rate codec can reduce the computational cost of speech language models by shortening the sequence length. Recent studies have developed 12.5Hz low-frame-rate audio codecs, but even lower frame rate codecs remain underexplored. We find that a major challenge for very low frame rate tokens is missing semantic information. This paper introduces FlexiCodec to address this limitation. FlexiCodec improves semantic preservation with a dynamic frame rate approach and introduces a novel architecture featuring an ASR feature-assisted dual stream encoding and Transformer bottlenecks. With dynamic frame rates, it uses less frames at information-sparse regions through adaptively merging semantically similar frames. A dynamic frame rate also allows FlexiCodec to support inference-time controllable frame rates between 3Hz and 12.5Hz. Experiments on 6.25Hz, 8.3Hz and 12.5Hz average frame rates confirm that FlexiCodec excels over baseline systems in semantic information preservation and delivers a high audio reconstruction quality. We also validate the effectiveness of FlexiCodec in language model-based TTS. Demos are available at: https://flexicodec.github.io. Code is available at: https://github.com/amphionteam/flexicodec.

音频编码动态帧率语音合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。