用2D扩散模型先验，让单光照3D场景实现真实光影重渲染。
A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis
- 利用2D扩散模型生成多光照图像，补足单光照数据
- 通过多层感知机控制光照方向，实现低频光照直接调节
- 适合需要真实3D重光照的场景重建与编辑任务
在多视角数据中，由于通常仅在单一光照条件下捕获，辐射场重光照问题严重欠约束，尤其对于包含多个物体的完整场景更为困难。本文提出一种方法，利用从2D图像扩散模型中提取的先验知识，仅基于单光照数据创建可重光照的辐射场。首先，在多光照数据集上微调2D扩散模型，使其根据光照方向条件生成图像，从而将单光照捕获扩展为具有现实感但可能存在不一致性的多光照数据。使用该增强数据训练由3D高斯点云表示的可重光照辐射场。为实现对低频光照方向的直接控制，采用以光照方向为输入的多层感知机表示外观；为保证多视角一致性并克服生成误差，引入每图像的辅助特征向量进行优化。在合成与真实多视角数据上的实验表明，该方法成功利用2D扩散模型先验，实现了完整场景的真实3D重光照。项目主页：https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/
原文摘要 · Abstract (English)
Relighting radiance fields is severely underconstrained for multi-view data, which is most often captured under a single illumination condition; It is especially hard for full scenes containing multiple objects. We introduce a method to create relightable radiance fields using such single-illumination data by exploiting priors extracted from 2D image diffusion models. We first fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by light direction, allowing us to augment a single-illumination capture into a realistic -- but possibly inconsistent -- multi-illumination dataset from directly defined light directions. We use this augmented data to create a relightable radiance field represented by 3D Gaussian splats. To allow direct control of light direction for low-frequency lighting, we represent appearance with a multi-layer perceptron parameterized on light direction. To enforce multi-view consistency and overcome inaccuracies we optimize a per-image auxiliary feature vector. We show results on synthetic and real multi-view data under single illumination, demonstrating that our method successfully exploits 2D diffusion model priors to allow realistic 3D relighting for complete scenes. Project site https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。