用高斯点阵实现虚拟制作中的可重布光3D重建，提升画面质量与编辑灵活性。
Relightable Gaussian Splatting for Virtual Production Using Image-Based Illumination
- 基于已知背景图像，将场景分解为固定外观与可变光照两部分。
- 通过纹理贴图采样模拟光照传输，无需物理渲染且支持任意输出变量。
- 实测训练<2小时，推理达35帧/秒，适合影视后期实时重布光需求。
虚拟制作（VP）使用LED墙同时提供背景图像和基于图像的照明。然而，这使光照与背景及场景外观绑定，限制了后期编辑的灵活性。传统逆向渲染依赖物理渲染估计三维几何与光照，使用环境贴图，但这些贴图通常分辨率低且假设远场光照。在近场、高分辨率的虚拟制作场景中，这会导致误差并增加编辑复杂性。为此，我们提出一种面向虚拟制作的高斯点阵3D重建与重布光框架。该框架利用已知背景图像来引导重布光过程，避免依赖环境贴图，将合成简化为背景图像编辑任务。为实现此框架，我们引入一个流程（及配套数据集），在不同背景内容与光照条件下捕获真实虚拟制作场景。该数据用于将3D场景分解为固定外观与可变光照组件。可变光照部分通过参数化每个基元的UV坐标、强度值与分辨率调节因子来模拟光传输。利用多级纹理图（mipmaps），直接在图像空间采样背景纹理——隐式捕捉反射与折射，无需物理渲染。结合固定外观组件，可使用高斯点阵光栅化器渲染重布光场景。相比基线方法，本方法实现更高精度3D重建与可控重布光。该方法高效（<3 GB内存，<5 GB显存，<2小时训练，~35 FPS），支持输出深度、光照强度、光照颜色、无光照渲染等任意变量。
原文摘要 · Abstract (English)
Virtual production (VP) use LED walls to provide both background imagery and image-based lighting. While this enables on-set compositing, it couples lighting to background and scene appearance, limiting flexibility for downstream editing. In addition, inverse rendering conventionally relies on physically-based rendering to estimates 3D geometry and lighting, using environment maps. However, these maps are typically low-resolution and assume far-field lighting. In VP, with near-field and high-resolution image-based lighting, this can lead to inaccuracies and introduce complexities when editing. Addressing this, we propose a VP-specific framework for 3D reconstruction and relighting using Gaussian Splatting. This uses the known background imagery to condition the relighting process. This avoids relying on environment maps and reduces compositing to a background-image editing task. To realize our framework, we introduce a process (and associated dataset) that captures real VP scenes under varying background content and illumination conditions. This data is used to decompose a 3D scene into fixed appearance and variable lighting components. The variable lighting process simulates light transport by parameterizing each primitive with a UV coordinate, intensity value and resolution modifier. Using mipmaps, these directly sample the background texture in image space - implicitly capturing reflections and refractions without physically-based rendering. Combined with the fixed appearance component, this allows us to render relit scenes using a Gaussian Splatting rasterizer. Compared to baselines, our approach achieves higher-quality 3D reconstruction and controllable relighting. The method is efficient (<3 GB RAM, <5 GB VRAM, <2 hours training, ~35 FPS) and supports rendering useful arbitrary output variables including depth, lighting intensity, lighting color, and unlit renders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。