综述语音大模型最新进展，解析端到端语音生成技术
Recent Advances in Speech Language Models: A Survey
- 提出端到端语音语言模型，跳过文本中间转换
- 解决多阶段转换导致的失真与延迟问题
- 适合语音交互、智能助手等场景研究者阅读
大型语言模型（LLMs）在文本交互中表现突出，但人类交流以语音为主，需发展语音驱动模型。传统方法采用ASR+LLM+TTS三阶段流水线，存在模态转换损失、延迟高和错误累积等问题。为此，语音语言模型（SpeechLMs）作为端到端语音生成方案应运而生。本文首次系统综述了SpeechLMs的构建方法，详述其架构关键组件与训练策略；系统梳理了其多样化能力，分类评估指标，并讨论该领域面临挑战与未来方向。相关资源已开源于GitHub。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have recently garnered significant attention, primarily for their capabilities in text-based interactions. However, natural human interaction often relies on speech, necessitating a shift towards voice-based models. A straightforward approach to achieve this involves a pipeline of ``Automatic Speech Recognition (ASR) + LLM + Text-to-Speech (TTS)", where input speech is transcribed to text, processed by an LLM, and then converted back to speech. Despite being straightforward, this method suffers from inherent limitations, such as information loss during modality conversion, significant latency due to the complex pipeline, and error accumulation across the three stages. To address these issues, Speech Language Models (SpeechLMs) -- end-to-end models that generate speech without converting from text -- have emerged as a promising alternative. This survey paper provides the first comprehensive overview of recent methodologies for constructing SpeechLMs, detailing the key components of their architecture and the various training recipes integral to their development. Additionally, we systematically survey the various capabilities of SpeechLMs, categorize their evaluation metrics, and discuss the challenges and future research directions in this rapidly evolving field. The GitHub repository is available at https://github.com/dreamtheater123/Awesome-SpeechLM-Survey
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。