让语音大模型直接理解语音，省去转写步骤，对话更流畅
Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM
- 不通过文字中间转换，直接在语音上推理
- 降低延迟，提升对语音的原生理解能力
- 适合实时语音交互系统研发者
当前基于语音的大型语言模型主要依赖大量ASR和TTS数据训练，在相关任务上表现良好，但在直接语音对话方面仍存在明显局限。这些模型通常采用ASR-to-TTS链式思维管道，先将语音转为文本处理，再生成音频回应，导致延迟增加并丢失音频特征。本文提出一种隐式内化ASR链式思维的方法，将语音理解能力直接嵌入语音大模型，减少延迟，增强模型对语音的原生理解能力，推动更高效自然的实时语音交互。同时，我们发布了大规模合成对话数据集以促进后续研究。
原文摘要 · Abstract (English)
Current speech-based LLMs are predominantly trained on extensive ASR and TTS datasets, excelling in tasks related to these domains. However, their ability to handle direct speech-to-speech conversations remains notably constrained. These models often rely on an ASR-to-TTS chain-of-thought pipeline, converting speech into text for processing before generating audio responses, which introduces latency and loses audio features. We propose a method that implicitly internalizes ASR chain of thought into a speech LLM, enhancing its native speech understanding capabilities. Our approach reduces latency and improves the model's native understanding of speech, paving the way for more efficient and natural real-time audio interactions. We also release a large-scale synthetic conversational dataset to facilitate further research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。