一次运行即可从无姿态视频帧重建3D场景并估计相机位姿
VicaSplat: A Single Run is All You Need for 3D Gaussian Splatting and Camera Estimation from Unposed Video Frames
- 基于变压器架构，用视觉令牌与可学习相机令牌协同建模
- 在多视角输入下超越基线方法，两视角表现媲美先进模型
- 跨数据集泛化能力强，ScanNet上无需微调即达最优
我们提出VicaSplat，一种从无姿态视频帧序列中联合进行3D高斯重建与相机位姿估计的新框架，该任务在真实世界3D应用中至关重要但研究不足。核心在于一种新型的Transformer网络架构：图像编码器将每张图像映射为一组视觉令牌，所有视觉令牌与额外插入的可学习相机令牌拼接后，通过定制化的Transformer解码器实现全量交互。相机令牌按因果顺序聚合来自不同视角的特征，并逐帧调制以注入视图依赖信息。随后通过不同的预测头估计3D高斯点和相机位姿参数。实验表明，VicaSplat在多视角输入下优于基线方法，且在两视角设置下表现媲美现有最优方案。尤为突出的是，其在ScanNet基准上展现出卓越的跨数据集泛化能力，无需任何微调即可取得领先性能。
原文摘要 · Abstract (English)
We present VicaSplat, a novel framework for joint 3D Gaussians reconstruction and camera pose estimation from a sequence of unposed video frames, which is a critical yet underexplored task in real-world 3D applications. The core of our method lies in a novel transformer-based network architecture. In particular, our model starts with an image encoder that maps each image to a list of visual tokens. All visual tokens are concatenated with additional inserted learnable camera tokens. The obtained tokens then fully communicate with each other within a tailored transformer decoder. The camera tokens causally aggregate features from visual tokens of different views, and further modulate them frame-wisely to inject view-dependent features. 3D Gaussian splats and camera pose parameters can then be estimated via different prediction heads. Experiments show that VicaSplat surpasses baseline methods for multi-view inputs, and achieves comparable performance to prior two-view approaches. Remarkably, VicaSplat also demonstrates exceptional cross-dataset generalization capability on the ScanNet benchmark, achieving superior performance without any fine-tuning. Project page: https://lizhiqi49.github.io/VicaSplat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。