用深度图+注意力机制,让图像修复更准更懂空间结构。
Leveraging Depth Maps and Attention Mechanisms for Enhanced Image Inpainting
- 双编码器分别处理彩色图和深度图,用注意力融合特征。
- 在线段和方块遮挡下,修复质量显著优于仅用彩色图的模型。
- 适合需要精准空间理解的图像修复任务,如场景重建。
现有基于深度学习的图像修复方法通常仅依赖RGB图像与卷积网络进行重构,但完全忽略深度信息可能导致对场景空间结构理解不足。如同人眼通过立体线索感知深度,将深度图引入修复过程可提升模型对上下文的理解能力。本文提出一种新方法,同时利用RGB图像与深度图进行图像修复。模型采用双编码器架构:一个处理RGB图像,另一个处理深度图;解码器中通过注意力机制融合两者的编码特征。我们采用线状和方块状两种掩码策略,测试模型在不同遮挡类型下的鲁棒性。为分析模型关注区域,使用梯度加权类激活映射(Grad-CAM)进行可视化。结果表明,结合深度信息能显著提升修复质量。定性与定量对比均显示,融合深度信息的模型优于基线,注意力机制进一步提升性能,多评估指标和可视化结果验证了其有效性。
原文摘要 · Abstract (English)
Existing deep learning-based image inpainting methods typically rely on convolutional networks with RGB images to reconstruct images. However, relying exclusively on RGB images may neglect important depth information, which plays a critical role in understanding the spatial and structural context of a scene. Just as human vision leverages stereo cues to perceive depth, incorporating depth maps into the inpainting process can enhance the model's ability to reconstruct images with greater accuracy and contextual awareness. In this paper, we propose a novel approach that incorporates both RGB and depth images for enhanced image inpainting. Our models employ a dual encoder architecture, where one encoder processes the RGB image and the other handles the depth image. The encoded features from both encoders are then fused in the decoder using an attention mechanism, effectively integrating the RGB and depth representations. We use two different masking strategies, line and square, to test the robustness of the model under different types of occlusions. To further analyze the effectiveness of our approach, we use Gradient-weighted Class Activation Mapping (Grad-CAM) visualizations to examine the regions of interest the model focuses on during inpainting. We show that incorporating depth information alongside the RGB image significantly improves the reconstruction quality. Through both qualitative and quantitative comparisons, we demonstrate that the depth-integrated model outperforms the baseline, with attention mechanisms further enhancing inpainting performance, as evidenced by multiple evaluation metrics and visualization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。