用大模型实现拼音到汉字的端到端转换，提升多语言语音识别效果。
LLM-based phoneme-to-grapheme for phoneme-based speech recognition
- 用大模型替代传统复杂流程，实现拼音到汉字的直接转换。
- 在波兰语和德语上分别降低3.6%和6.9%的错误率。
- 适合追求高精度多语言语音识别的研究者与开发者。
在自动语音识别中，基于音素的多语言预训练与跨语言微调因其高效的数据利用和媲美子词模型的表现而备受关注。然而，基于加权有限状态转换器（WFST）的解码受限于复杂流程且难以利用大语言模型（LLM）。为此，本文提出基于大模型的音素到字形（LLM-P2G）解码方法，包含语音到音素（S2P）和音素到字形（P2G）两阶段。挑战在于级联S2P与P2G可能造成信息丢失。为此，我们提出两种训练策略：带噪声音素的数据增强（DANP）和随机化Top-K边缘化（TKM）训练与解码。实验结果表明，在波兰语和德语的跨语言语音识别任务中，LLM-P2G相比WFST系统分别实现3.6%和6.9%的相对词错误率下降。
原文摘要 · Abstract (English)
In automatic speech recognition (ASR), phoneme-based multilingual pre-training and crosslingual fine-tuning is attractive for its high data efficiency and competitive results compared to subword-based models. However, Weighted Finite State Transducer (WFST) based decoding is limited by its complex pipeline and inability to leverage large language models (LLMs). Therefore, we propose LLM-based phoneme-to-grapheme (LLM-P2G) decoding for phoneme-based ASR, consisting of speech-to-phoneme (S2P) and phoneme-to-grapheme (P2G). A challenge is that there seems to have information loss in cascading S2P and P2G. To address this challenge, we propose two training strategies: data augmentation with noisy phonemes (DANP), and randomized top-$K$ marginalized (TKM) training and decoding. Our experimental results show that LLM-P2G outperforms WFST-based systems in crosslingual ASR for Polish and German, by relative WER reductions of 3.6% and 6.9% respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。