一秒钟重建可随意调光的3D模型,打破传统流程分割瓶颈。
ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination
- 用多视角交叉条件变换器融合输入,实现几何、材质、光照统一建模。
- 在真实与合成数据混合训练下,重建精度显著提升,光照解耦更准确。
- 适合需要快速生成可调光3D资产的工业设计与虚拟现实场景。
从图像重建3D资产长期依赖几何、材质和光照分别处理的独立流程,各自存在局限且计算开销大。我们提出ReLi3D,首个端到端统一管道,仅需不到一秒即可从稀疏多视角图像中同时重建完整3D几何、空间变化的物理材质及环境光照。核心洞察是多视角约束能极大改善材质与光照解耦,而单图方法仍面临根本性病态问题。关键在于采用变压器跨条件融合多视图输入,并结合新颖的双路径预测策略:第一条路径预测物体结构与外观,第二条路径从背景或反射预测环境光照。配合可微分的蒙特卡洛多重重要性采样渲染器,构建了最优光照解耦训练流程。此外,通过混合域训练协议(结合合成PBR数据集与真实RGB采集),实现了在几何、材质准确率和光照质量上的泛化性能。通过将以往分离任务整合为一次前向传播,实现近实时生成完整可调光3D资产。
原文摘要 · Abstract (English)
Reconstructing 3D assets from images has long required separate pipelines for geometry reconstruction, material estimation, and illumination recovery, each with distinct limitations and computational overhead. We present ReLi3D, the first unified end-to-end pipeline that simultaneously reconstructs complete 3D geometry, spatially-varying physically-based materials, and environment illumination from sparse multi-view images in under one second. Our key insight is that multi-view constraints can dramatically improve material and illumination disentanglement, a problem that remains fundamentally ill-posed for single-image methods. Key to our approach is the fusion of the multi-view input via a transformer cross-conditioning architecture, followed by a novel unified two-path prediction strategy. The first path predicts the object's structure and appearance, while the second path predicts the environment illumination from image background or object reflections. This, combined with a differentiable Monte Carlo multiple importance sampling renderer, creates an optimal illumination disentanglement training pipeline. In addition, with our mixed domain training protocol, which combines synthetic PBR datasets with real-world RGB captures, we establish generalizable results in geometry, material accuracy, and illumination quality. By unifying previously separate reconstruction tasks into a single feed-forward pass, we enable near-instantaneous generation of complete, relightable 3D assets. Project Page: https://reli3d.jdihlmann.com/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。