重新设计语音编码器,让压缩后的语音标记更适配语言模型生成。
Rethinking Speech Codecs: From Compression to Autoregressive Generative Modeling

- 训练时加入自回归兼容约束,提升语音标记的时序一致性。
- 不同层采用异构下采样,区分语义与声学信息,增强对齐效果。
- 在多个数据集上验证有效,适合现有语言模型继续预训练语音数据。
近期语音语言模型利用预训练编码器生成的离散语音表示,实现可扩展的训练与生成。然而,现有编码器主要针对压缩优化,未考虑语言模型训练的自回归特性,导致压缩后语音标记建模性能不佳。本文从生成建模角度重新审视语音离散化,提出一种新框架,显式对齐语音标记化与自回归训练。该方法在编码器训练中引入自回归兼容约束,鼓励具有时间一致性和可预测性的标记序列。此外,提出针对不同层的异构下采样策略,区分语义与声学层,提升语义标记与对应文本内容的对齐。多基准实验表明,该方法弥合了语音压缩与生成建模之间的差距,显著提升现有语言模型在语音数据上的持续预训练效果。结果在多个编码器上一致改进,验证了方法的通用性与适用性。
原文摘要 · Abstract (English)
Recent advances in speech language models leverage discrete speech representations from pretrained codecs to enable scalable training and generation. However, existing codecs are primarily optimized for compression without accounting for the autoregressive nature of language model training, resulting in suboptimal performance when modeling compressed speech tokens. In this work, we revisit speech discretization from a generative modeling perspective and propose a novel framework that explicitly aligns speech tokenization with autoregressive training. Our approach introduces autoregressive-compatible constraints during codec training, encouraging token sequences that exhibit temporal consistency and predictability. In addition, we propose a heterogeneous downsampling strategy for different layers of speech tokens, distinguishing semantic from acoustic layers, to improve the alignment between semantic tokens and corresponding textual content. Extensive experiments across multiple benchmarks demonstrate that our method bridges the gap between speech compression and generative modeling, enabling more effective continued pretraining of existing language models on speech data. The approach consistently improves performance across multiple codecs, validating its generality and applicability to diverse speech modeling scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。