将物理渲染与高斯建模结合,实现可重光照的高质量动态三维视频。
BEAM: Bridging Physically-based Rendering and Gaussian Modeling for Relightable Volumetric Video
- 用分阶段优化恢复几何与材质属性,融合多视角信息。
- 通过扩散模型生成粗糙度,2D到3D策略推导环境光遮蔽和基础色。
- 输出可适配实时与离线渲染的可重光照资产,适合影视与交互应用。
体素视频通过捕捉动态三维场景,为虚拟现实、教育和远程呈现等应用提供沉浸式体验。然而,传统方法受限于固定光照,而神经方法在效率、质量和可重光照适应性之间存在权衡。为此,我们提出BEAM,一种将4D高斯表示与物理基于渲染(PBR)结合的新流程,仅需多视角RGB视频即可生成高质量、可重光照的体素视频。BEAM通过一系列高斯技术恢复详细几何与PBR属性:首先在粗到精优化框架中融合高斯人体动作追踪与几何感知光栅化,恢复时空一致的几何结构;随后逐步增强高斯属性——利用多视角条件扩散模型生成粗糙度,并采用2D到3D策略推导环境光遮蔽(AO)与基础色,结合定制高斯射线追踪器高效计算可见性。恢复后的动态可重光照资产可无缝集成至传统计算机图形管线,支持实时延迟着色与离线光线追踪渲染。通过在多种光照条件下生成逼真自然的视觉效果,BEAM为互动娱乐、叙事创作与创意可视化开辟新可能。
原文摘要 · Abstract (English)
Volumetric video enables immersive experiences by capturing dynamic 3D scenes, enabling diverse applications for virtual reality, education, and telepresence. However, traditional methods struggle with fixed lighting conditions, while neural approaches face trade-offs in efficiency, quality, or adaptability for relightable scenarios. To address these limitations, we present BEAM, a novel pipeline that bridges 4D Gaussian representations with physically-based rendering (PBR) to produce high-quality, relightable volumetric videos from multi-view RGB footage. BEAM recovers detailed geometry and PBR properties via a series of available Gaussian-based techniques. It first combines Gaussian-based human performance tracking with geometry-aware rasterization in a coarse-to-fine optimization framework to recover spatially and temporally consistent geometries. We further enhance Gaussian attributes by incorporating PBR properties step by step. We generate roughness via a multi-view-conditioned diffusion model, and then derive AO and base color using a 2D-to-3D strategy, incorporating a tailored Gaussian-based ray tracer for efficient visibility computation. Once recovered, these dynamic, relightable assets integrate seamlessly into traditional CG pipelines, supporting real-time rendering with deferred shading and offline rendering with ray tracing. By offering realistic, lifelike visualizations under diverse lighting conditions, BEAM opens new possibilities for interactive entertainment, storytelling, and creative visualization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。