自研U-Net与Transformer实现多语言图像文字翻译
A U-Net and Transformer Pipeline for Multilingual Image Translation
- 用自定义U-Net检测图像中文本区域，结合Tesseract识别
- 从零训练多语言Seq2Seq Transformer，支持5种语言翻译
- 全定制化流程，适合需要可控、可调试的翻译系统
本文提出一个端到端的多语言图像文字翻译流水线，集成自定义U-Net进行文本检测、Tesseract引擎进行文本识别，以及从零训练的序列到序列（Seq2Seq）Transformer模型用于神经机器翻译（NMT）。该方法首先使用在合成数据集上训练的U-Net模型，准确分割并检测图像中的文本区域；这些区域随后由Tesseract提取源文本；提取的文本输入至在跨5种语言的平行语料库上从零训练的自定义Transformer模型中进行翻译。与依赖单一预训练模型的系统不同，本架构强调完全定制化和可扩展性。系统在文本检测准确率、文本识别质量及翻译性能（以BLEU分数衡量）方面进行评估，整体流程表现良好，验证了构建专用系统直接从图像翻译文字的可行性。
原文摘要 · Abstract (English)
This paper presents an end-to-end multilingual translation pipeline that integrates a custom U-Net for text detection, the Tesseract engine for text recognition, and a from-scratch sequence-to-sequence (Seq2Seq) Transformer for Neural Machine Translation (NMT). Our approach first utilizes a U-Net model, trained on a synthetic dataset , to accurately segment and detect text regions from an image. These detected regions are then processed by Tesseract to extract the source text. This extracted text is fed into a custom Transformer model trained from scratch on a multilingual parallel corpus spanning 5 languages. Unlike systems reliant on monolithic pre-trained models, our architecture emphasizes full customization and adaptability. The system is evaluated on its text detection accuracy, text recognition quality, and translation performance via BLEU scores. The complete pipeline demonstrates promising results, validating the viability of a custom-built system for translating text directly from images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。