让音频编码器更懂语言，提升语音模型生成质量
LLM-Codec: Neural Audio Codec Meets Language Model Objectives
- 用语言模型目标增强编码器训练，保持原有结构不变
- 语音连贯性提升12.1点，困惑度降低35%
- 适合做语音生成、语音识别的模型开发者
神经音频编码器广泛用作语音语言模型的分词器，但其训练目标是波形重建而非自回归预测，导致离散令牌空间中引入声学不确定性，增加语言模型困惑度。本文提出 ours，通过引入面向语言模型的目标来增强编码器训练，同时保持编码器和大语言模型架构不变。方法包括：(i) 使用类似 Medusa 的多步预测头，提升多步可预测性；(ii) 通过内存池对比损失实现音频与文本表征的语义对齐。可微分 Gumbel 桥使这些目标的梯度能端到端传递至编码器。在 SALMon 语音连贯性任务上，基于 ours 的令牌语言模型达到 61.6% 准确率（较 AUV 提升 12.1 点），困惑度降低 35%。在 Codec-SUPERB-tiny 上， ours 将语音梅尔距离降低 5.0%，同时实现学习性能提升，证明重建保真度与令牌可预测性可协同优化。
原文摘要 · Abstract (English)
Neural audio codecs are widely used as tokenizers for spoken language models, but they are optimized for waveform reconstruction rather than autoregressive prediction. This mismatch injects acoustically driven uncertainty into the discrete token space and increases language-model perplexity. We propose \ours, which augments codec training with language-model-facing objectives while keeping both codec and LLM architectures unchanged. \ours introduces (i) future token prediction with Medusa-style multi-step heads to encourage multi-step predictability, and (ii) semantic alignment that matches audio and text representations via a memory-bank contrastive loss. A differentiable Gumbel bridge enables end-to-end gradients from these objectives to the codec encoder. On SALMon speech coherence, token LMs trained on \ours reach 61.6% accuracy (+12.1 points over AUV) while reducing perplexity 35. On Codec-SUPERB-tiny, \ours improves speech Mel distance by 5.0% over AUV while simultaneously achieving the learnability gains, demonstrating that reconstruction fidelity and token predictability can be improved together.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。