对比三类图文翻译模型，发现多模态大模型表现最佳。
Comparative Evaluation of Machine Translation Systems on Images with Text
- 分三类对比：模块化流水线、多模态大模型、端到端图像翻译
- 多模态大模型在多个语言对上综合性能最优，超越模块化和端到端方案
- 适合关注跨模态理解与多语言生成的研究者参考
本文对应用于含文本图像的机器翻译系统进行了比较评估，该任务融合了计算机视觉与自然语言处理。研究对比了三种主要范式：将文本检测、识别与翻译分离的模块化流水线；能联合处理图像与文本的多模态大语言模型（MLLMs）；以及直接生成翻译后图像的端到端模型Translatotron-V。模块化系统采用最先进的OCR工具docTR结合多语言LLM如Llama和EuroLLM，MLLMs则包含不同配置的Gemini 2.5。实验在涵盖多种语言对的并行多语言数据集上进行，使用BLEU、chrF和TER指标评估。结果表明，模块化流水线优于端到端方法，而多模态大模型整体表现最佳，展现出更强的灵活性与上下文理解能力。这些发现凸显了多模态推理在图像到文本翻译中的有效性，为未来多语言环境下视觉理解与语言生成融合研究提供了坚实基础。
原文摘要 · Abstract (English)
This work presents a comparative evaluation of machine translation systems applied to images containing textual information, a task that lies at the intersection of computer vision and natural language processing. The study compares three main paradigms: modular pipelines that separate text detection, recognition, and translation; multi-modal large language models (MLLMs) capable of processing both image and text jointly; and an end-to-end model, Translatotron-V, which directly generates translated images. The modular systems employ state-of-the-art OCR (docTR) combined with multilingual LLMs such as Llama and EuroLLM, while the evaluated MLLMs include different configurations of Gemini 2.5. Experiments were conducted on parallel multilingual datasets covering multiple language pairs, with evaluation based on BLEU, chrF, and TER metrics. The results show that modular pipelines outperform the end-to-end approach, while MLLMs achieve the best overall performance, demonstrating superior flexibility and contextual understanding. These findings underscore the effectiveness of multi-modal reasoning for image-to-text translation and provide a solid foundation for future research on integrating visual understanding and language generation in multilingual settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。