让大模型同步自检OCR能力，提升文档图像翻译效果
Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
- 训练时先让模型自动生成OCR文本，再翻译，强化双任务协同
- 在OCR和翻译任务上均显著减少遗忘，通用性更强
- 适合需要兼顾文字识别与跨语言翻译的文档处理场景
多模态大模型（MLLM）在文档图像任务中表现优异，尤其在光学字符识别（OCR）方面。然而，在文档图像机器翻译（DIMT）任务中，模型面临跨模态与跨语言双重挑战。以往通过在DIMT数据集上监督微调（SFT）的方法常导致模型遗忘原有的单语能力，如OCR性能。为此，我们提出一种新型微调范式——同步自检（SSR）OCR能力，受‘双语认知优势’启发。具体而言，SSR在生成翻译前先要求模型输出OCR文本，使模型能利用其强大的单语OCR能力的同时学习跨语言翻译。大量实验证明，该方法有效缓解灾难性遗忘，显著提升MLLM在OCR与DIMT任务上的泛化能力。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model's existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage". Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。