arXiv:2511.05229cs.CVcs.AI2025-11NeurIPS被引 2

无需已知相机位姿,实现单目视频中动态场景的高质量重建与渲染。

4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos

  • 分两阶段解耦静态与动态成分,先用3D基础模型估计初始位姿和几何,再进行运动感知优化。
  • 在真实动态数据集上相比顶尖方法提升1.8dB PSNR,大动态物体场景表现更优。
  • 适合做动态场景重建、实时渲染或缺乏相机位姿信息的研究者使用。

从未知相机位姿的单目视频中实现动态场景的新视角合成仍是计算机视觉与图形学中的核心挑战。尽管近期基于神经辐射场(NeRF)和3D高斯点阵(3DGS)的3D表示方法在静态场景中取得显著进展,但对动态内容处理能力有限,且通常依赖预计算的相机位姿。本文提出4D3R,一种无位姿依赖的动态神经渲染框架,通过两阶段策略解耦静态与动态成分。首先利用3D基础模型进行初始位姿与几何估计,随后进行运动感知精修。4D3R引入两项关键技术:(1) 运动感知捆绑调整(MA-BA)模块,融合基于Transformer的学习先验与SAM2,实现鲁棒的动态物体分割,提升相机位姿精修精度;(2) 高效的运动感知高斯点阵(MA-GS)表示,采用带形变场MLP的控制点与线性混合皮肤技术建模动态运动,在保持高质量重建的同时将计算开销降低5倍。大量实验证明,该方法在真实动态数据集上相比当前最优方法提升最高达1.8dB PSNR,尤其在大范围动态物体场景中优势明显。

原文摘要 · Abstract (English)

Novel view synthesis from monocular videos of dynamic scenes with unknown camera poses remains a fundamental challenge in computer vision and graphics. While recent advances in 3D representations such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have shown promising results for static scenes, they struggle with dynamic content and typically rely on pre-computed camera poses. We present 4D3R, a pose-free dynamic neural rendering framework that decouples static and dynamic components through a two-stage approach. Our method first leverages 3D foundational models for initial pose and geometry estimation, followed by motion-aware refinement. 4D3R introduces two key technical innovations: (1) a motion-aware bundle adjustment (MA-BA) module that combines transformer-based learned priors with SAM2 for robust dynamic object segmentation, enabling more accurate camera pose refinement; and (2) an efficient Motion-Aware Gaussian Splatting (MA-GS) representation that uses control points with a deformation field MLP and linear blend skinning to model dynamic motion, significantly reducing computational cost while maintaining high-quality reconstruction. Extensive experiments on real-world dynamic datasets demonstrate that our approach achieves up to 1.8dB PSNR improvement over state-of-the-art methods, particularly in challenging scenarios with large dynamic objects, while reducing computational requirements by 5x compared to previous dynamic scene representations.

动态重建神经渲染单目视频运动建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。