提出结构纹理协同重建机制，解决卷积下采样中的信息丢失问题
Thinking inside the Convolution for Image Inpainting: Reconstructing Texture via Structure under Global and Local Side
- 设计统计归一化与反归一化策略，实现结构与纹理特征的双向互补
- 在256×256和512×512图像上均超越现有方法，尤其替换编码器后提升显著
- 适合关注图像修复中高频细节恢复的研究者与开发者
图像修复近年来取得显著进展，主要依赖编码器-解码器架构，利用卷积神经网络（CNN）通过卷积下采样从已知区域语义推断遮挡区域，并经解码器上采样生成最终结果。现有方法通常认为CNN能提取高频结构与低频纹理，但忽略了下采样过程中结构与纹理特征图的信息损失，导致上采样输出不理想。本文系统探讨了结构与纹理特征是否及如何在下采样阶段相互帮助以缓解信息损失。针对结构与纹理特征图，提出统计归一化与反归一化策略，作为下采样过程中的重建引导。大量实验验证其优势：在256×256与512×512分辨率图像上均优于当前最优方法，尤其当全部编码器被替换为本方法时效果更佳。代码已开源。
原文摘要 · Abstract (English)
Image inpainting has earned substantial progress, owing to the encoder-and-decoder pipeline, which is benefited from the Convolutional Neural Networks (CNNs) with convolutional downsampling to inpaint the masked regions semantically from the known regions within the encoder, coupled with an upsampling process from the decoder for final inpainting output. Recent studies intuitively identify the high-frequency structure and low-frequency texture to be extracted by CNNs from the encoder, and subsequently for a desirable upsampling recovery. However, the existing arts inevitably overlook the information loss for both structure and texture feature maps during the convolutional downsampling process, hence suffer from a non-ideal upsampling output. In this paper, we systematically answer whether and how the structure and texture feature map can mutually help to alleviate the information loss during the convolutional downsampling. Given the structure and texture feature maps, we adopt the statistical normalization and denormalization strategy for the reconstruction guidance during the convolutional downsampling process. The extensive experimental results validate its advantages to the state-of-the-arts over the images from low-to-high resolutions including 256*256 and 512*512, especially holds by substituting all the encoders by ours. Our code is available at https://github.com/htyjers/ConvInpaint-TSGL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。