用单目视频训练4D高斯变换器，实现动态场景秒级重建
4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos
- 以4D高斯为先验，统一建模静态与动态物体
- 处理64帧滚动窗口，实时渲染且支持长时序输入
- 纯前向推理替代优化方法，适合真实世界视频应用
我们提出4DGT，一种基于4D高斯的Transformer模型，完全在真实世界单目位姿视频上训练，用于动态场景重建。通过4D高斯作为归纳偏置，4DGT统一建模静态与动态成分，可刻画具有不同物体生命周期的复杂时变环境。我们设计了一种新颖的密度控制策略，使模型能处理更长的时空输入，并保持运行时高效渲染。模型以滚动窗口方式处理64个连续位姿帧，预测场景中一致的4D高斯分布。不同于基于优化的方法，4DGT采用纯前向推理，将重建时间从小时级降至秒级，且可扩展至长视频序列。仅在大规模单目位姿视频数据集上训练，4DGT在真实世界视频上显著优于以往高斯基网络，在跨域视频上达到与优化方法相当的精度。
原文摘要 · Abstract (English)
We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We proposed a novel density control strategy in training, which enables our 4DGT to handle longer space-time input and remain efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can outperform prior Gaussian-based networks significantly in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos. Project page: https://4dgt.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。