用思维链训练语音对话系统，让模型更懂对话逻辑。
Chain-of-Thought Training for Open E2E Spoken Dialogue Systems
- 引入思维链机制，让模型逐步推理对话逻辑。
- 仅用300小时公开数据，ROUGE-1提升1.5以上。
- 适合想高效训练语音对话系统的研究者。
与传统级联流水线不同，端到端（E2E）语音对话系统保持全程可微，能捕捉非语音信息，更适合建模口语交互。但现有方法通常需要大规模数据，且生成回复语义不连贯。本文提出一种简单有效的方法，利用思维链（CoT）形式，使对话数据训练紧密衔接多模态语言模型在语音识别（ASR）、文本转语音合成（TTS）和文本语言模型任务上的预训练。该方法在公开人类对话语料（如Switchboard）上仅需300小时数据即可成功训练，相比基线实现超过1.5的ROUGE-1提升，且计算效率高。模型与训练代码将公开发布。
原文摘要 · Abstract (English)
Unlike traditional cascaded pipelines, end-to-end (E2E) spoken dialogue systems preserve full differentiability and capture non-phonemic information, making them well-suited for modeling spoken interactions. However, existing E2E approaches often require large-scale training data and generates responses lacking semantic coherence. We propose a simple yet effective strategy leveraging a chain-of-thought (CoT) formulation, ensuring that training on conversational data remains closely aligned with the multimodal language model (LM)'s pre-training on speech recognition~(ASR), text-to-speech synthesis (TTS), and text LM tasks. Our method achieves over 1.5 ROUGE-1 improvement over the baseline, successfully training spoken dialogue systems on publicly available human-human conversation datasets, while being compute-efficient enough to train on just 300 hours of public human-human conversation data, such as the Switchboard. We will publicly release our models and training code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。