用拼音嵌入预训练提升大模型语音识别能力
Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs
- 让大模型先学拼音转汉字，再处理真实语音
- 在AISHELL-1上相对基线提升9.5%准确率
- 适合做中文语音识别与多模态融合的研究者
将大型语言模型（LLMs）与预训练语音模型结合，为自动语音识别（ASR）开辟了新路径。尽管LLMs在多模态理解任务中表现优异，但有效利用其能力提升ASR仍面临挑战。本文提出一种新训练方法，通过在拼音嵌入序列（代表发音特征）上预训练LLMs以生成对应汉字，使模型在接触真实语音数据前即可适应从发音特征生成文本。此外，通过微调LoRA参数增强模型对语音模态信息的理解。在AISHELL-1数据集上，该方法相比无拼音到汉字预训练的基线，实现了9.5%的相对性能提升；若在拼音到汉字预训练中引入辅助文本数据，性能进一步提升至19.0%的相对改进。
原文摘要 · Abstract (English)
The integration of large language models (LLMs) with pre-trained speech models has opened up new avenues in automatic speech recognition (ASR). While LLMs excel in multimodal understanding tasks, effectively leveraging their capabilities for ASR remains a significant challenge. This paper presents a novel training approach to enhance LLM performance in ASR tasks. We propose pre-training LLMs on Pinyin embedding sequences, which represent pronunciation features, to generate corresponding Chinese characters. This step enables the LLM to adapt to generating text from pronunciation features before encountering real speech data. Furthermore, we fine-tune the LoRA parameters to enhance the LLM's understanding of speech modality information. In AISHELL-1 corpus, our approach yields a 9.5% relative improvement in ASR tasks compared to the baseline without Pinyi-to-Character pre-training. Additionally, incorporating auxiliary text data for Pinyi-to-Character pre-training further boosts performance, achieving a 19.0% relative improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。