无需目标语言数据，用大模型实现跨语言语音识别
Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations
- 通过罗马音预测学习通用语音表示，摆脱语言依赖
- 在82种语言上实现零样本语音识别，跨语言泛化能力强
- 适合需要快速支持新语言的语音识别系统开发者
本文提出一种新型零样本视听语音识别框架Zero-AVSR，可在无目标语言音视频数据的情况下实现跨语言语音识别。核心是引入音频-视觉语音罗马化器（AV-Romanizer），通过预测罗马字母来学习语言无关的语音表示；再利用大语言模型（LLM）强大的多语言能力，将罗马文转换为具体语言字符，构建级联式零样本识别系统。进一步提出统一框架，直接将AV-Romanizer编码的视听表示注入LLM，通过多任务微调适配。为此构建了包含82种语言、总计2,916小时音视频数据的多语言视听罗马化语料库（MARC），提供每种语言的本体文字与罗马音转写。大量实验表明，该框架能有效扩展至训练时未见的语言，具备强大跨语言泛化能力。
原文摘要 · Abstract (English)
We explore a novel zero-shot Audio-Visual Speech Recognition (AVSR) framework, dubbed Zero-AVSR, which enables speech recognition in target languages without requiring any audio-visual speech data in those languages. Specifically, we introduce the Audio-Visual Speech Romanizer (AV-Romanizer), which learns language-agnostic speech representations by predicting Roman text. Then, by leveraging the strong multilingual modeling capabilities of Large Language Models (LLMs), we propose converting the predicted Roman text into language-specific graphemes, forming the proposed Cascaded Zero-AVSR. Taking it a step further, we explore a unified Zero-AVSR approach by directly integrating the audio-visual speech representations encoded by the AV-Romanizer into the LLM. This is achieved through finetuning the adapter and the LLM using our proposed multi-task learning scheme. To capture the wide spectrum of phonetic and linguistic diversity, we also introduce a Multilingual Audio-Visual Romanized Corpus (MARC) consisting of 2,916 hours of audio-visual speech data across 82 languages, along with transcriptions in both language-specific graphemes and Roman text. Extensive analysis and experiments confirm that the proposed Zero-AVSR framework has the potential to expand language support beyond the languages seen during the training of the AV-Romanizer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。