用课程学习让大模型更高效地实现多语种语音转文字翻译
Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning
- 分三阶段课程学习，利用大模型的翻译能力迁移至语音任务
- 在15×14个语言对上达当前最佳，每语言少于10小时语音即可训练
- 适合资源稀缺语种的语音转写，开源代码与模型
多模态大语言模型（MLLM）在语音转文字翻译（S2TT）任务中取得显著进展。现有研究多聚焦英语为中心的翻译方向，而多对多翻译受限于平行数据稀缺。为此，我们提出一种三阶段课程学习策略，利用大语言模型的机器翻译能力并适配至S2TT任务，实现在低资源环境下的有效学习。我们在不同参数规模（3B、7B、32B）的MLLM上进行训练，并使用FLEURS和CoVoST-2数据集评估该策略。实验表明，该方法在15×14个语言对上达到当前最优平均性能，每语言仅需不到10小时语音数据即可获得竞争力结果。源代码与模型已公开于https://github.com/yxduir/LLM-SRT。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved significant success in Speech-to-Text Translation (S2TT) tasks. While most existing research has focused on English-centric translation directions, the exploration of many-to-many translation is still limited by the scarcity of parallel data. To address this, we propose a three-stage curriculum learning strategy that leverages the machine translation capabilities of large language models and adapts them to S2TT tasks, enabling effective learning in low-resource settings. We trained MLLMs with varying parameter sizes (3B, 7B, and 32B) and evaluated the proposed strategy using the FLEURS and CoVoST-2 datasets. Experimental results show that the proposed strategy achieves state-of-the-art average performance in $15\times14$ language pairs, requiring fewer than 10 hours of speech data per language to achieve competitive results. The source code and models are released at https://github.com/yxduir/LLM-SRT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。