提出高效语音文本联合解码新方法，提升对话系统响应速度与准确性
Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model
- 采用统一模型对比交错与并行生成解码策略
- 交错解码对齐效果最佳，但推理慢
- 新提出的早停交错法加速解码且性能略优
语音语言模型（Speech LMs）在单一模型中实现端到端语音-文本建模，为语音对话系统提供了有前景的方向。解码策略的选择对性能、效率和对齐质量至关重要。本文在相同基础语言模型、语音分词器和训练数据的受控环境下，系统比较了代表性的语音-文本联合解码策略，包括交错式与并行生成模式。结果表明，交错式方法对齐效果最佳，但因序列过长导致推理缓慢。为此，我们提出一种新型早停交错（ESI）模式，显著加快解码速度，并取得略优性能。此外，我们构建了高质量问答数据集，进一步提升语音问答表现。
原文摘要 · Abstract (English)
Speech language models (Speech LMs) enable end-to-end speech-text modeling within a single model, offering a promising direction for spoken dialogue systems. The choice of speech-text jointly decoding paradigm plays a critical role in performance, efficiency, and alignment quality. In this work, we systematically compare representative joint speech-text decoding strategies, including the interleaved, and parallel generation paradigms, under a controlled experimental setup using the same base language model, speech tokenizer and training data. Our results show that the interleaved approach achieves the best alignment. However it suffers from slow inference due to long token sequence length. To address this, we propose a novel early-stop interleaved (ESI) pattern that not only significantly accelerates decoding but also yields slightly better performance. Additionally, we curate high-quality question answering (QA) datasets to further improve speech QA performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。