用时间注意力机制提升动态3D高斯的运动建模能力
TimeFormer: Capturing Temporal Relationships of Deformable 3D Gaussians for Robust Reconstruction
- 引入跨时序变换器，隐式学习高斯点的运动关系
- 在复杂运动场景下重建精度显著提升，尤其在剧烈运动时
- 可插拔设计，推理速度不受影响，适合现有框架升级
动态场景重建是3D视觉中的长期挑战。近期方法通过添加形变场并将运动流等显式约束引入可变形3D高斯点云以实现动态重建，但这些方法独立学习各时间戳的运动变化，难以应对剧烈运动、极端几何或反射表面等复杂情况。为此，本文提出一个即插即用模块TimeFormer，使现有可变形3D高斯方法具备从学习角度隐式建模运动模式的能力。TimeFormer包含一个跨时序变换器编码器，自适应学习可变形3D高斯点之间的时序关系。此外，提出双流优化策略，在训练阶段将TimeFormer学到的运动知识迁移至主流水线，从而在推理时移除TimeFormer，保持原有渲染速度。大量多视角与单目动态场景实验验证了TimeFormer在定性和定量上的显著改进。
原文摘要 · Abstract (English)
Dynamic scene reconstruction is a long-term challenge in 3D vision. Recent methods extend 3D Gaussian Splatting to dynamic scenes via additional deformation fields and apply explicit constraints like motion flow to guide the deformation. However, they learn motion changes from individual timestamps independently, making it challenging to reconstruct complex scenes, particularly when dealing with violent movement, extreme-shaped geometries, or reflective surfaces. To address the above issue, we design a plug-and-play module called TimeFormer to enable existing deformable 3D Gaussians reconstruction methods with the ability to implicitly model motion patterns from a learning perspective. Specifically, TimeFormer includes a Cross-Temporal Transformer Encoder, which adaptively learns the temporal relationships of deformable 3D Gaussians. Furthermore, we propose a two-stream optimization strategy that transfers the motion knowledge learned from TimeFormer to the base stream during the training phase. This allows us to remove TimeFormer during inference, thereby preserving the original rendering speed. Extensive experiments in the multi-view and monocular dynamic scenes validate qualitative and quantitative improvement brought by TimeFormer. Project Page: https://patrickddj.github.io/TimeFormer/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。