用多模块协同修复模糊文本,减少幻觉,提升真实文本还原度。
Unified Diffusion Transformer for High-fidelity Text-Aware Image Restoration
- 融合扩散模型、视觉语言模型与文本检测模块,迭代优化文本恢复
- 在SA-Text和Real-Text上实现最优端到端F1分数,显著降低文本幻觉
- 适合需要高保真文本重建的图像修复场景,如历史文档恢复
文本感知图像修复(TAIR)旨在从包含退化文本内容的低质量输入中恢复高质量图像。尽管扩散模型在通用图像修复中具备强大的生成先验,但在以文本为中心的任务中常因缺乏显式语言知识而产生文本幻觉。为此,我们提出UniT,一种统一的文本修复框架,通过迭代方式整合扩散变压器(DiT)、视觉语言模型(VLM)和文本检测模块(TSM)。在UniT中,VLM从退化图像中提取文本内容,提供显式文本引导;同时,基于扩散特征训练的TSM在每个去噪步骤生成中间OCR预测,使VLM能迭代优化其引导。最终,利用强大表征能力的DiT骨干网络,结合这些线索恢复细粒度文本内容,并有效抑制文本幻觉。在SA-Text和Real-Text基准上的实验表明,UniT能忠实重建退化文本,显著减少幻觉,实现TAIR任务的端到端最优F1分数。
原文摘要 · Abstract (English)
Text-Aware Image Restoration (TAIR) aims to recover high-quality images from low-quality inputs containing degraded textual content. While diffusion models provide strong generative priors for general image restoration, they often produce text hallucinations in text-centric tasks due to the absence of explicit linguistic knowledge. To address this, we propose UniT, a unified text restoration framework that integrates a Diffusion Transformer (DiT), a Vision-Language Model (VLM), and a Text Spotting Module (TSM) in an iterative fashion for high-fidelity text restoration. In UniT, the VLM extracts textual content from degraded images to provide explicit textual guidance. Simultaneously, the TSM, trained on diffusion features, generates intermediate OCR predictions at each denoising step, enabling the VLM to iteratively refine its guidance during the denoising process. Finally, the DiT backbone, leveraging its strong representational power, exploit these cues to recover fine-grained textual content while effectively suppressing text hallucinations. Experiments on the SA-Text and Real-Text benchmarks demonstrate that UniT faithfully reconstructs degraded text, substantially reduces hallucinations, and achieves state-of-the-art end-to-end F1-score performance in TAIR task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。