让语音模型同时学语言和声音，生成更自然的语音。
Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modeling
- 用连续向量联合建模语义与声学信息，替代传统分步生成。
- 预测多个未来语义标记，提升语言信息保留能力。
- 适合需要精细声学控制的语音生成场景。
无文本监督的语音语言模型（SLMs）是不依赖文本标注的语音生成模型。现有大多数文本无关的SLMs通过预测下一个语义标记（即语言内容的离散表示）来建模语音，并依赖独立声码器为生成语音添加声学细节。这类模型缺乏声学上下文感知能力，也难以控制声学特征。本文提出一种新方法，通过同时生成语义标记与连续实值声学帧表示，联合建模语言与声学信息。采用流匹配目标，以语义标记为条件预测连续向量。我们探索了该方法的设计空间，发现预测多个未来语义标记有助于保持语言信息。实验表明，该方法在语言似然基准上表现与现有模型相当，同时在提示生成中提供更优的声学细节。
原文摘要 · Abstract (English)
Textless spoken language models (SLMs) are generative models of speech that do not rely on text supervision. Most textless SLMs learn to predict the next semantic token, a discrete representation of linguistic content, and rely on a separate vocoder to add acoustic information to the generated speech. Such models have no access to acoustic context and no built-in control over acoustic details. In this work, we propose to jointly model linguistic and acoustic information by generating semantic tokens and a continuous real-valued representation of the acoustic frame. We use a flow-matching objective to predict the continuous vector conditioned on the semantic tokens. We study the design space of this approach and find that predicting multiple future semantic tokens helps preserve linguistic information. Our approach achieves comparable performance to existing models in terms of linguistic likelihood benchmarks, while providing better acoustic detail in prompted generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。