提出高效多模态翻译训练框架，实现语音与文本翻译同步优化。
EMMeTT: Efficient Multimodal Machine Translation Training
- 设计联合训练机制，平衡语言、数据集与模态采样。
- 在FLORES和FLEURS上，四语言任务表现优于基线模型。
- 适合需同时支持语音与文本翻译的研究者与开发者。
基础语言模型的模态扩展引发对高效多模态训练方法的关注。本文聚焦神经机器翻译（NMT），提出一种融合自动语音翻译（AST）的语音-大语言模型联合训练方案。研究对比了两种架构：仅解码器的GPT与编码器-解码器的T5，均扩展了Canary-1B的语音编码器。为提升训练效率，提出新型训练框架EMMeTT，包含跨语言、数据集与模态的均衡采样、高效顺序数据遍历及创新的2D分桶策略，并配合批大小优化器（OOMptimizer）。实验表明，多模态训练对两类架构均有提升。使用EMMeTT训练的SALM-T5在保留原始NMT能力的同时，在FLORES和FLEURS的四语言子集上超越现有AST基线。最终模型可同时生成高质量文本与语音翻译结果。
原文摘要 · Abstract (English)
A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST). We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder. To handle joint multimodal training, we propose a novel training framework called EMMeTT. EMMeTT improves training efficiency with the following: balanced sampling across languages, datasets, and modalities; efficient sequential data iteration; and a novel 2D bucketing scheme for multimodal data, complemented by a batch size optimizer (OOMptimizer). We show that a multimodal training consistently helps with both architectures. Moreover, SALM-T5 trained with EMMeTT retains the original NMT capability while outperforming AST baselines on four-language subsets of FLORES and FLEURS. The resultant Multimodal Translation Model produces strong text and speech translation results at the same time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。