提出方向感知的动态场景重建方法,提升复杂运动下的渲染质量。
DaRePlane: Direction-aware Representations for Dynamic Scene Reconstruction
- 从六个方向学习动态场景的方向感知表示,结合双树复小波变换恢复平面信息。
- 在NeRF和高斯溅射中均实现先进性能,复杂动态场景下新视角合成效果最佳。
- 引入可学习掩码减少冗余,降低存储开销,适合医疗等复杂动态场景应用。
近年来,基于平面的显式表示被广泛用于建模与重渲染动态场景,以缓解神经辐射场(NeRF)和高斯溅射(GS)训练慢的问题。然而,仅将4D动态场景分解为多个2D平面表示难以实现复杂运动下的高保真重渲染。为此,我们提出DaRePlane,一种方向感知的表示方法,通过六种不同方向捕捉场景动态。该表示经逆双树复小波变换(DTCWT)恢复平面信息。在NeRF流程中,DaRePlane通过融合这些恢复平面的向量,生成时空点特征,再由小型MLP进行颜色回归;在高斯溅射中,则对高斯点计算特征,并通过小型多头MLP预测时空形变。为解决六组实部与六组虚部方向感知小波系数带来的冗余问题,我们引入可学习掩码机制,在不显著损失性能的前提下减轻存储负担。我们在常规与手术动态场景上,分别测试了其在NeRF与GS系统中的表现。大量实验表明,DaRePlane在多种复杂动态场景的新视角合成任务中达到当前最优水平。
原文摘要 · Abstract (English)
Numerous recent approaches to modeling and re-rendering dynamic scenes leverage plane-based explicit representations, addressing slow training times associated with models like neural radiance fields (NeRF) and Gaussian splatting (GS). However, merely decomposing 4D dynamic scenes into multiple 2D plane-based representations is insufficient for high-fidelity re-rendering of scenes with complex motions. In response, we present DaRePlane, a novel direction-aware representation approach that captures scene dynamics from six different directions. This learned representation undergoes an inverse dual-tree complex wavelet transformation (DTCWT) to recover plane-based information. Within NeRF pipelines, DaRePlane computes features for each space-time point by fusing vectors from these recovered planes, then passed to a tiny MLP for color regression. When applied to Gaussian splatting, DaRePlane computes the features of Gaussian points, followed by a tiny multi-head MLP for spatial-time deformation prediction. Notably, to address redundancy introduced by the six real and six imaginary direction-aware wavelet coefficients, we introduce a trainable masking approach, mitigating storage issues without significant performance decline. To demonstrate the generality and efficiency of DaRePlane, we test it on both regular and surgical dynamic scenes, for both NeRF and GS systems. Extensive experiments show that DaRePlane yields state-of-the-art performance in novel view synthesis for various complex dynamic scenes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。