arXiv:2503.08387cs.CV2025-03CVPR被引 11

让文字识别模型和编辑任务协同工作,提升场景文本修改效果。

Recognition-Synergistic Scene Text Editing

  • 将识别与编辑统一在单一框架中,利用识别模型隐式分离风格与内容。
  • 在合成和真实数据集上均达到当前最优性能,真实场景效果显著。
  • 无需配对数据即可训练,适合实际应用中缺乏标注的情况。

场景文本编辑旨在修改图像中的文字内容同时保持风格一致。传统方法通过显式分离源图像的风格与内容,再融合目标内容并用预训练识别模型保证内容一致性,但流程复杂,在复杂场景下表现不佳。本文提出一种新型方法——识别协同场景文本编辑(RS-STE),充分利用文本识别的内在协同能力。模型在统一框架中集成识别与编辑,借助识别模型隐式实现风格与内容解耦,并保障内容一致性。具体采用基于Transformer的多模态并行解码器,同时预测文本内容与风格化图像;引入循环自监督微调策略,仅用未配对的真实世界数据即可训练,通过双循环生成过程增强风格与内容一致性。基于较简单架构,RS-STE在合成与真实世界基准上均达当前最优性能,并进一步验证了生成困难样本对下游识别任务的提升作用。代码已公开于 https://github.com/ZhengyaoFang/RS-STE。

原文摘要 · Abstract (English)

Scene text editing aims to modify text content within scene images while maintaining style consistency. Traditional methods achieve this by explicitly disentangling style and content from the source image and then fusing the style with the target content, while ensuring content consistency using a pre-trained recognition model. Despite notable progress, these methods suffer from complex pipelines, leading to suboptimal performance in complex scenarios. In this work, we introduce Recognition-Synergistic Scene Text Editing (RS-STE), a novel approach that fully exploits the intrinsic synergy of text recognition for editing. Our model seamlessly integrates text recognition with text editing within a unified framework, and leverages the recognition model's ability to implicitly disentangle style and content while ensuring content consistency. Specifically, our approach employs a multi-modal parallel decoder based on transformer architecture, which predicts both text content and stylized images in parallel. Additionally, our cyclic self-supervised fine-tuning strategy enables effective training on unpaired real-world data without ground truth, enhancing style and content consistency through a twice-cyclic generation process. Built on a relatively simple architecture, RS-STE achieves state-of-the-art performance on both synthetic and real-world benchmarks, and further demonstrates the effectiveness of leveraging the generated hard cases to boost the performance of downstream recognition tasks. Code is available at https://github.com/ZhengyaoFang/RS-STE.

文本编辑图像生成识别协同自监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。