arXiv:2409.18770cs.CV2024-09中稿 · publication as a R…被引 7

基于物理约束的单图重光照方法,提升真实感与泛化能力

Relighting from a Single Image: Datasets and Deep Intrinsic-based Architecture

  • 分两阶段利用图像内在成分分解,中间输出引入物理一致性约束
  • 在自建合成与真实数据集上均超越现有最先进方法,尤其在任意光照下表现优异
  • 适用于需要高质量重光照的动画生成场景,开源数据与代码

单图场景重光照旨在生成一个新版本的输入图像,使其看起来像是被新的目标光照条件照射。尽管已有工作从多个角度探索该问题,但在任意光照条件下生成逼真重光照图像仍极具挑战性,且相关数据集稀缺。本文从数据集和方法两方面入手,提出两个新数据集:一个包含内在成分真实值的合成数据集,以及在实验室条件下采集的真实数据集,缓解了现有数据匮乏问题。为在重光照流程中引入物理一致性,我们建立了一个基于内在分解的两阶段网络,中间步骤输出可施加物理约束。当训练集缺乏内在分解真实标签时,引入无监督模块确保内在输出质量。所提方法在现有及自建数据集上均优于当前最先进方法。此外,使用本研究合成数据集进行预训练可提升其他方法在其他数据集上的性能。由于方法可适配任意光照条件,具备生成动画序列的能力。数据集、代码与视频均已公开。

原文摘要 · Abstract (English)

Single image scene relighting aims to generate a realistic new version of an input image so that it appears to be illuminated by a new target light condition. Although existing works have explored this problem from various perspectives, generating relit images under arbitrary light conditions remains highly challenging, and related datasets are scarce. Our work addresses this problem from both the dataset and methodological perspectives. We propose two new datasets: a synthetic dataset with the ground truth of intrinsic components and a real dataset collected under laboratory conditions. These datasets alleviate the scarcity of existing datasets. To incorporate physical consistency in the relighting pipeline, we establish a two-stage network based on intrinsic decomposition, giving outputs at intermediate steps, thereby introducing physical constraints. When the training set lacks ground truth for intrinsic decomposition, we introduce an unsupervised module to ensure that the intrinsic outputs are satisfactory. Our method outperforms the state-of-the-art methods in performance, as tested on both existing datasets and our newly developed datasets. Furthermore, pretraining our method or other prior methods using our synthetic dataset can enhance their performance on other datasets. Since our method can accommodate any light conditions, it is capable of producing animated results. The dataset, method, and videos are publicly available.

重光照图像生成内在分解物理约束

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。