用参考图修复时间不同但场景相似的损坏图像,效果优于现有方法。
Time-variant Image Inpainting via Interactive Distribution Transition Estimation
- 通过交互式分布迁移估计,自适应融合两张时间不同的图像信息。
- 在新构建的TAMP-Street数据集上,恢复精度显著高于现有顶尖方法。
- 适合需要跨时间修复图像的场景,如老照片或监控视频修复。
本文聚焦于一项新颖且实用的任务——时间可变图像修复(TAMP),目标是利用一张与目标图像存在显著时间差的参考图像,恢复受损的目标图像。与传统参考引导修复不同,TAMP下的参考图像不仅内容差异大,还可能自身也存在损伤。该任务在日常生活中常见,例如通过一张旧照片修复另一张损坏的照片,但参考图像来源和质量无法保证。我们发现,即使最先进的参考引导修复方法在此任务中仍因图像信息混乱而表现不佳。为此,提出交互式分布迁移估计(InDiTE)模块,通过自适应语义互补,促进受损区域的恢复。进一步提出InDiTE-Diff方法,将InDiTE与先进扩散模型结合,并在采样阶段引入潜在空间跨参考机制。此外,由于缺乏相关基准,我们基于现有图像与掩码数据集构建了新数据集TAMP-Street。在两种不同时间差异设置下,在TAMP-Street上的实验表明,我们的方法持续优于现有最先进参考引导图像修复方法。
原文摘要 · Abstract (English)
In this work, we focus on a novel and practical task, i.e., Time-vAriant iMage inPainting (TAMP). The aim of TAMP is to restore a damaged target image by leveraging the complementary information from a reference image, where both images captured the same scene but with a significant time gap in between, i.e., time-variant images. Different from conventional reference-guided image inpainting, the reference image under TAMP setup presents significant content distinction to the target image and potentially also suffers from damages. Such an application frequently happens in our daily lives to restore a damaged image by referring to another reference image, where there is no guarantee of the reference image's source and quality. In particular, our study finds that even state-of-the-art (SOTA) reference-guided image inpainting methods fail to achieve plausible results due to the chaotic image complementation. To address such an ill-posed problem, we propose a novel Interactive Distribution Transition Estimation (InDiTE) module which interactively complements the time-variant images with adaptive semantics thus facilitate the restoration of damaged regions. To further boost the performance, we propose our TAMP solution, namely Interactive Distribution Transition Estimation-driven Diffusion (InDiTE-Diff), which integrates InDiTE with SOTA diffusion model and conducts latent cross-reference during sampling. Moreover, considering the lack of benchmarks for TAMP task, we newly assembled a dataset, i.e., TAMP-Street, based on existing image and mask datasets. We conduct experiments on the TAMP-Street datasets under two different time-variant image inpainting settings, which show our method consistently outperform SOTA reference-guided image inpainting methods for solving TAMP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。