无需语音合成，用语义表示训练语音指令模型
Speechless: Speech Instruction Training Without Speech for Low Resource Languages
- 在语义层面生成指令，跳过语音合成环节
- 仅用文本指令微调大模型，仍能理解语音输入
- 适合语音数据稀缺的语言资源建设
大型语言模型驱动的语音助手快速发展，但高质量语音指令数据稀缺，尤其对低资源语言。现有合成语音需依赖优质文本到语音（TTS）模型，而此类模型常不可用。本文提出新方法，在语义表征层面停止合成，直接对齐预训练的Whisper编码器，使大模型仅通过文本指令即可完成微调，推理时仍能理解语音指令。该方法简化了训练流程，为低资源语言语音助手构建提供了可行路径。
原文摘要 · Abstract (English)
The rapid growth of voice assistants powered by large language models (LLM) has highlighted a need for speech instruction data to train these systems. Despite the abundance of speech recognition data, there is a notable scarcity of speech instruction data, which is essential for fine-tuning models to understand and execute spoken commands. Generating high-quality synthetic speech requires a good text-to-speech (TTS) model, which may not be available to low resource languages. Our novel approach addresses this challenge by halting synthesis at the semantic representation level, bypassing the need for TTS. We achieve this by aligning synthetic semantic representations with the pre-trained Whisper encoder, enabling an LLM to be fine-tuned on text instructions while maintaining the ability to understand spoken instructions during inference. This simplified training process is a promising approach to building voice assistant for low-resource languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。