用大模型生成对话文本，再合成自然语音，提升口语对话连贯性。
SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation
- 先用大模型写对话内容，再转成音素序列并预测发音时长
- 在Fisher数据集上生成的语音语义连贯性显著提升
- 适合需要高质量口语生成的应用场景
近期基于语音单元的「无文本」语音语言模型（SLMs）在生成自然语音方面取得显著进展，包括非语言发声。然而生成的语音常缺乏语义连贯性。本文提出一种语音语言模型与大语言模型融合的自发口语对话生成框架（SLIDE）。首先利用大语言模型生成对话文本，随后将文本转换为音素序列，并采用双塔Transformer结构的时长预测器预测每个音素的持续时间。最后，以语音音素序列为条件，使用语音语言模型对对话进行发声合成。在Fisher数据集上的实验表明，该系统能生成自然且语义连贯的口语对话。
原文摘要 · Abstract (English)
Recently, ``textless" speech language models (SLMs) based on speech units have made huge progress in generating naturalistic speech, including non-verbal vocalizations. However, the generated speech samples often lack semantic coherence. In this paper, we propose SLM and LLM Integration for spontaneous spoken Dialogue gEneration (SLIDE). Specifically, we first utilize an LLM to generate the textual content of spoken dialogue. Next, we convert the textual dialogues into phoneme sequences and use a two-tower transformer-based duration predictor to predict the duration of each phoneme. Finally, an SLM conditioned on the spoken phoneme sequences is used to vocalize the textual dialogue. Experimental results on the Fisher dataset demonstrate that our system can generate naturalistic spoken dialogue while maintaining high semantic coherence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。