用语音和文本融合提升多语言翻译,无需真实语音数据。
Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion
- 用自生成语音+文本融合输入大模型,突破图像数据限制。
- 在Multi30K和FLORES-200上均达新最优,108个方向表现领先。
- 合成语音与真实语音效果接近,适合资源匮乏语言对。
多模态大语言模型通过融合多模态信息显著提升了翻译性能,但现有研究主要依赖图像引导方法,受限于多语言图文对的稀缺性。语音模态因与文本天然对齐且数据丰富,更易实现大规模语言覆盖。本文提出一种语音引导机器翻译框架(SMT),将语音与文本作为融合输入注入多模态大模型以提升翻译质量。为减少对低资源数据的依赖,引入自演化机制:包含文本转语音模型生成合成语音,以及能分类合成样本并基于正样本迭代优化自身的多模态大模型。实验表明,该框架在Multi30K多模态翻译基准上超越所有现有方法,达到新最佳结果;在通用翻译数据集FLORES-200上,108个翻译方向平均表现达到当前最优。在CoVoST-2上的消融实验确认,合成语音与真实语音差异对翻译质量影响可忽略。代码与模型已开源:https://github.com/yxduir/LLM-SRT。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved notable success in enhancing translation performance by integrating multimodal information. However, existing research primarily focuses on image-guided methods, whose applicability is constrained by the scarcity of multilingual image-text pairs. The speech modality overcomes this limitation due to its natural alignment with text and the abundance of existing speech datasets, which enable scalable language coverage. In this paper, we propose a Speech-guided Machine Translation (SMT) framework that integrates speech and text as fused inputs into an MLLM to improve translation quality. To mitigate reliance on low-resource data, we introduce a Self-Evolution Mechanism. The core components of this framework include a text-to-speech model, responsible for generating synthetic speech, and an MLLM capable of classifying synthetic speech samples and iteratively optimizing itself using positive samples. Experimental results demonstrate that our framework surpasses all existing methods on the Multi30K multimodal machine translation benchmark, achieving new state-of-the-art results. Furthermore, on general machine translation datasets, particularly the FLORES-200, it achieves average state-of-the-art performance in 108 translation directions. Ablation studies on CoVoST-2 confirms that differences between synthetic and authentic speech have negligible impact on translation quality. The code and models are released at https://github.com/yxduir/LLM-SRT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。