用变分框架自动学习语音韵律特征，提升语音生成自然度。
A Variational Framework for Improving Naturalness in Generative Spoken Language Models
- 通过变分自编码器自动学习语音的连续属性特征
- 生成语音在自然度上显著优于仅依赖音高特征的方法
- 适合关注语音合成自然性与端到端建模的研究者
大型语言模型在文本处理中的成功激发了其向语音建模的迁移。然而，由于语音是连续且复杂的，通常需离散化以进行自回归建模。由自监督模型生成的语义标记（semantic tokens）主要关注语言内容，却忽略了韵律信息，导致基于这些标记训练的模型生成的语音缺乏自然性。现有方法尝试通过添加音高特征来弥补，但音高无法全面表征语音中的副语言属性，且特征选择依赖人工精心设计。为此，我们提出一种端到端的变分框架，可自动学习并编码这些连续语音属性以增强语义标记。该方法无需手动提取和筛选副语言特征，且生成结果更受人类评估者青睐。代码、样本与模型已公开于 https://github.com/b04901014/vae-gslm。
原文摘要 · Abstract (English)
The success of large language models in text processing has inspired their adaptation to speech modeling. However, since speech is continuous and complex, it is often discretized for autoregressive modeling. Speech tokens derived from self-supervised models (known as semantic tokens) typically focus on the linguistic aspects of speech but neglect prosodic information. As a result, models trained on these tokens can generate speech with reduced naturalness. Existing approaches try to fix this by adding pitch features to the semantic tokens. However, pitch alone cannot fully represent the range of paralinguistic attributes, and selecting the right features requires careful hand-engineering. To overcome this, we propose an end-to-end variational approach that automatically learns to encode these continuous speech attributes to enhance the semantic tokens. Our approach eliminates the need for manual extraction and selection of paralinguistic features. Moreover, it produces preferred speech continuations according to human raters. Code, samples and models are available at https://github.com/b04901014/vae-gslm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。