用大模型实现零样本语音翻译与识别，无需配对数据
Zero-resource Speech Translation and Recognition with LLMs
- 用多语言大模型+语音编码器+轻量适配模块，打通无配对数据的语言
- 语音翻译在新语言上达BLEU 23以上，语音识别误识率低至28.2%
- 突破点在于依赖大模型语言生成能力，适合无标注数据场景
尽管语音处理技术不断进步，零资源语音翻译（ST）和自动语音识别（ASR）仍是难题。本文提出利用多语言大语言模型（LLM）在从未见过配对音频-文本数据的语言上进行ST和ASR。方法包括预训练的多语言语音编码器、多语言LLM以及一个将音频表示映射到LLM词嵌入空间的轻量适配模块。我们在ST和ASR任务上进行了多项实验，探究最佳训练策略及对未见语言性能影响最大的数据类型。在语音翻译中，最优模型在CoVoST2上对两种未见语言的BLEU分数超过23；在语音识别中，最高达到28.2%的词错误率（WER）。最后表明，系统性能受限于LLM生成目标语言文本的能力。
原文摘要 · Abstract (English)
Despite recent advancements in speech processing, zero-resource speech translation (ST) and automatic speech recognition (ASR) remain challenging problems. In this work, we propose to leverage a multilingual Large Language Model (LLM) to perform ST and ASR in languages for which the model has never seen paired audio-text data. We achieve this by using a pre-trained multilingual speech encoder, a multilingual LLM, and a lightweight adaptation module that maps the audio representations to the token embedding space of the LLM. We perform several experiments both in ST and ASR to understand how to best train the model and what data has the most impact on performance in previously unseen languages. In ST, our best model is capable to achieve BLEU scores over 23 in CoVoST2 for two previously unseen languages, while in ASR, we achieve WERs of up to 28.2\%. We finally show that the performance of our system is bounded by the ability of the LLM to output text in the desired language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。