用文本框标签预训练文本去除模型，降低标注成本。
Leveraging Text Localization for Scene Text Removal via Text-aware Masked Image Modeling
- 设计双流架构，利用文本框标签实现弱监督预训练。
- 在SCUT-EnsText上达37.35 PSNR，性能领先。
- 适合缺乏像素级标注数据的研究者使用。
现有场景文本去除（STR）任务受限于昂贵的像素级标注数据。本文提出文本感知的掩码图像建模算法（TMIM），可仅用低成本的文本检测标签（如文本边界框）预训练STR模型。与以往依赖间接辅助任务提升隐式特征提取的方法不同，TMIM首次实现了STR任务的直接弱监督训练，显式高效地挖掘去除知识。具体而言，背景建模分支通过恢复被掩码的非文本区域，学习背景生成规律，并为掩码文本区域生成伪标签；文本擦除分支则基于伪标签学习端到端的文本去除能力。两个分支协同工作，使模型仅使用公开文本检测数据集即可取得优异性能，显著缓解高成本标注的瓶颈。实验表明，该方法优于其他预训练方法，在SCUT-EnsText上达到37.35 PSNR，代码将开源。
原文摘要 · Abstract (English)
Existing scene text removal (STR) task suffers from insufficient training data due to the expensive pixel-level labeling. In this paper, we aim to address this issue by introducing a Text-aware Masked Image Modeling algorithm (TMIM), which can pretrain STR models with low-cost text detection labels (e.g., text bounding box). Different from previous pretraining methods that use indirect auxiliary tasks only to enhance the implicit feature extraction ability, our TMIM first enables the STR task to be directly trained in a weakly supervised manner, which explores the STR knowledge explicitly and efficiently. In TMIM, first, a Background Modeling stream is built to learn background generation rules by recovering the masked non-text region. Meanwhile, it provides pseudo STR labels on the masked text region. Second, a Text Erasing stream is proposed to learn from the pseudo labels and equip the model with end-to-end STR ability. Benefiting from the two collaborative streams, our STR model can achieve impressive performance only with the public text detection datasets, which greatly alleviates the limitation of the high-cost STR labels. Experiments demonstrate that our method outperforms other pretrain methods and achieves state-of-the-art performance (37.35 PSNR on SCUT-EnsText). Code will be available at https://github.com/wzx99/TMIM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。