让语音转文字翻译支持70种语言，且速度更快。
MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages
- 用课程学习和数据均衡扩展到70种语言的多对多翻译
- 将语音序列压缩至30个标记，推理速度显著提升
- 适合需要大规模多语言语音翻译的开发者和研究者
多模态大语言模型（MLLMs）在语音转文本翻译（S2TT）任务中表现优异，但现有研究受限于语言覆盖范围和效率问题。多数主流S2TT数据集以英语为主，限制了MLLMs多对多翻译能力的扩展；此外，当语音转换为长序列（如750个标记）时，MLLMs的推理速度急剧下降。为此，我们提出多语言低成本加速语音转文字翻译框架MCAT，包含两项创新：第一，采用课程学习与数据平衡策略，将语言覆盖扩展至70种，并实现这些语言间的互译；第二，设计优化的语音适配器模块，将语音序列长度压缩至仅30个标记。在不同规模的MLLM（9B和27B）上进行的大量实验表明，MCAT不仅在FLEURS数据集上70×69个方向上超越当前最优端到端模型，还显著提升推理效率。代码与模型已开源：https://github.com/yxduir/m2m-70。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved great success in Speech-to-Text Translation (S2TT) tasks. However, current research is constrained by two key challenges: language coverage and efficiency. Most of the popular S2TT datasets are substantially English-centric, which restricts the scaling-up of MLLMs' many-to-many translation capabilities. Moreover, the inference speed of MLLMs degrades dramatically when the speech is converted into long sequences (e.g., 750 tokens). To address these limitations, we propose a Multilingual Cost-effective Accelerated Speech-to-Text Translator (MCAT) framework, which includes two innovations. First, a language scaling method that leverages curriculum learning and a data balancing strategy is introduced to extend the language coverage supported by MLLMs to 70 languages and achieve mutual translation among these languages. Second, an optimized speech adapter module is designed to reduce the length of the speech sequence to only 30 tokens. Extensive experiments were conducted on MLLMs of different scales (9B and 27B). The experimental results demonstrate that MCAT not only surpasses state-of-the-art end-to-end models on the FLEURS dataset across 70x69 directions but also enhances inference efficiency. The code and models are released at https://github.com/yxduir/m2m-70.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。