用ViT生成补丁替换掩码区，提升图像修复质量
Image inpainting enhancement by replacing the original mask with a self-attended region from the input image
- 用ViT从图像中自注意力生成补丁替代原掩码区域
- 在4个公开数据集上显著优于4种标准模型
- 可直接使用预训练ViT，无需额外训练
图像修复是通过重建像素信息来恢复图像缺失或损坏区域的过程，近年来深度学习方法取得了显著进展。本文提出一种基于视觉变换器（ViT）的新型深度学习预处理方法，用于图像修复。该方法将掩码区域的像素值替换为由ViT生成的像素，利用注意力矩阵中的多样视觉补丁捕捉判别性空间特征。据我们所知，这是首个针对图像修复任务提出的此类预处理模型。此外，实验表明该方法可有效结合预训练的ViT模型与预设补丁大小使用。为评估通用性，我们在四个公开数据集上对比了四种标准模型，结果证明该预处理技术能显著提升修复性能。
原文摘要 · Abstract (English)
Image inpainting, the process of restoring missing or corrupted regions of an image by reconstructing pixel information, has recently seen considerable advancements through deep learning-based approaches. In this paper, we introduce a novel deep learning-based pre-processing methodology for image inpainting utilizing the Vision Transformer (ViT). Our approach involves replacing masked pixel values with those generated by the ViT, leveraging diverse visual patches within the attention matrix to capture discriminative spatial features. To the best of our knowledge, this is the first instance of such a pre-processing model being proposed for image inpainting tasks. Furthermore, we show that our methodology can be effectively applied using the pre-trained ViT model with pre-defined patch size. To evaluate the generalization capability of the proposed methodology, we provide experimental results comparing our approach with four standard models across four public datasets, demonstrating the efficacy of our pre-processing technique in enhancing inpainting performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。