用单流方式训练语音语言模型，无需文本监督即可生成自然语音。
WavSLM: Single-Stream Speech Language Modeling via WavLM Distillation
- 将自监督语音表示量化到统一码本，用自回归预测训练单流模型。
- 在一致性评测中表现媲美主流方法，参数更少、数据需求更低。
- 适合追求高效、可流式推理的语音生成场景，如实时对话系统。
大型语言模型表明，简单的自回归训练可实现可扩展且连贯的生成，但将这一范式拓展至语音仍面临语义与声学信息纠缠的挑战。现有语音语言模型多依赖文本监督、分层标记流或复杂混合架构，偏离了在文本领域已被证实有效的单流生成预训练范式。本文提出 WavSLM，通过将自监督的 WavLM 表示量化并蒸馏至单一码本，并优化自回归下一区块预测目标进行训练。WavSLM 在不使用文本监督或文本预训练的前提下，于单个标记流中联合建模语义与声学信息。尽管结构简单，其在一致性评测和语音生成任务上表现优异，且参数更少、训练数据更少，同时支持流式推理。
原文摘要 · Abstract (English)
Large language models show that simple autoregressive training can yield scalable and coherent generation, but extending this paradigm to speech remains challenging due to the entanglement of semantic and acoustic information. Most existing speech language models rely on text supervision, hierarchical token streams, or complex hybrid architectures, departing from the single-stream generative pretraining paradigm that has proven effective in text. In this work, we introduce WavSLM, a speech language model trained by quantizing and distilling self-supervised WavLM representations into a single codebook and optimizing an autoregressive next-chunk prediction objective. WavSLM jointly models semantic and acoustic information within a single token stream without text supervision or text pretraining. Despite its simplicity, it achieves competitive performance on consistency benchmarks and speech generation while using fewer parameters, less training data, and supporting streaming inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。