3B模型通过单阶段联合训练，实现语音与文本的高效融合。
VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning
- 用单阶段联合微调，在低秩适配器上同时训练语音与文本能力。
- 3B模型在语音基准测试中超越7B/13B模型，且不丢弃原有文本能力。
- 能处理多轮、混合模态等新任务，具备涌现能力，适合语音交互场景。
近期研究将大语言模型（LLM）扩展至语音能力，催生了语音语言模型（SpeechLM）。早期SpeechLM仅支持单轮语音问答，用户输入为语音上下文加文本问题。近年研究拓展至多轮对话，但通常需复杂多阶段监督微调（SFT）和多样数据。另一关键挑战是灾难性遗忘：优化语音任务的模型会严重退化文本性能。为此，我们提出一种新颖的单阶段联合语音-文本SFT方法，基于LLM主干的低秩适配（LoRA）。该联合微调结合纯文本数据与三类语音相关数据：语音识别与翻译、语音问答、多模态SFT。相较于7B或13B参数的先前SpeechLM，我们的3B模型在多个语音基准测试中表现更优，同时保持原始文本任务能力。此外，模型展现出有效处理未见提示与任务的涌现能力，包括多轮与混合模态输入。
原文摘要 · Abstract (English)
Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models (SpeechLMs). Earlier SpeechLMs focused on single-turn speech-based question answering (QA), where user input comprised a speech context and a text question. More recent studies have extended this to multi-turn conversations, though they often require complex, multi-stage supervised fine-tuning (SFT) with diverse data. Another critical challenge with SpeechLMs is catastrophic forgetting, where models optimized for speech tasks suffer significant degradation in text-only performance. To mitigate these issues, we propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the LLM backbone. Our joint SFT combines text-only SFT data with three types of speech-related data: speech recognition and translation, speech-based QA, and mixed-modal SFT. Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks while preserving the original capabilities on text-only tasks. Furthermore, our model shows emergent abilities of effectively handling previously unseen prompts and tasks, including multi-turn, mixed-modal inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。