提出新模型DebackX，解决真实背景下的图像内文本翻译难题
Exploring In-Image Machine Translation with Real-World Background
- 分离图像背景与文本区域，直接对文本区域翻译
- 在真实背景图像上翻译效果提升，视觉一致性更好
- 适合需要真实场景文本翻译的应用，如多语言导航
图像内机器翻译（IIMT）旨在将图像中的文本从一种语言翻译成另一种语言。以往研究多集中在简化的场景，如单行黑色文字位于白色背景上的图像，与真实世界相差甚远，实用性有限。为使IIMT研究更具现实价值，需考虑文本背景源自真实图像的复杂场景。为此，我们构建了一个包含真实背景字幕文本的IIMT数据集。然而，现有IIMT模型在复杂场景中表现不佳。为此，我们提出DebackX模型：先将源图像中的背景与文本图像分离，直接对文本图像进行翻译，再将翻译后的文本图像融合回原始背景，生成目标图像。实验结果表明，该模型在翻译质量与视觉效果上均取得显著提升。
原文摘要 · Abstract (English)
In-Image Machine Translation (IIMT) aims to translate texts within images from one language to another. Previous research on IIMT was primarily conducted on simplified scenarios such as images of one-line text with black font in white backgrounds, which is far from reality and impractical for applications in the real world. To make IIMT research practically valuable, it is essential to consider a complex scenario where the text backgrounds are derived from real-world images. To facilitate research of complex scenario IIMT, we design an IIMT dataset that includes subtitle text with real-world background. However previous IIMT models perform inadequately in complex scenarios. To address the issue, we propose the DebackX model, which separates the background and text-image from the source image, performs translation on text-image directly, and fuses the translated text-image with the background, to generate the target image. Experimental results show that our model achieves improvements in both translation quality and visual effect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。