让3D高斯点阵模型无需姿态信息,直接处理视角不佳的输入。
UFV-Splatter: Pose-Free Feed-Forward 3D Gaussian Splatting Adapted to Unfavorable Views
- 用重中心化图像+低秩适配层,让预训练模型适应新视角。
- 引入高斯适配模块与对齐方法,提升几何一致性与渲染精度。
- 适合真实场景中相机位姿未知的3D重建任务。
本文提出一种无姿态、前馈式3D高斯点阵框架,可处理视角不利的输入。现有前馈方法通常在世界原点放置3D物体,并从指向原点的相机视角(即有利视角)进行渲染,限制了其在真实场景中相机位姿变化或未知情况下的应用。为突破此限制,我们设计了一种新适配框架,使预训练的无姿态前馈3DGS模型能有效应对不利视角。通过将重中心化图像输入预训练模型并结合低秩适配(LoRA)层,利用有利视角中学习到的先验知识。进一步提出高斯适配模块以增强重中心输入生成的高斯几何一致性,并引入高斯对齐方法,实现准确的目标视角渲染用于训练。此外,采用仅含有利视角图像的现成数据集进行训练。在来自Google Scanned Objects的合成数据和OmniObject3D的真实数据上的实验验证了该方法在处理不利输入视角时的有效性。
原文摘要 · Abstract (English)
This paper presents a pose-free, feed-forward 3D Gaussian Splatting (3DGS) framework designed to handle unfavorable input views. A common rendering setup for training feed-forward approaches places a 3D object at the world origin and renders it from cameras pointed toward the origin -- i.e., from favorable views, limiting the applicability of these models to real-world scenarios involving varying and unknown camera poses. To overcome this limitation, we introduce a novel adaptation framework that enables pretrained pose-free feed-forward 3DGS models to handle unfavorable views. We leverage priors learned from favorable images by feeding recentered images into a pretrained model augmented with low-rank adaptation (LoRA) layers. We further propose a Gaussian adapter module to enhance the geometric consistency of the Gaussians derived from the recentered inputs, along with a Gaussian alignment method to render accurate target views for training. Additionally, we introduce a new training strategy that utilizes an off-the-shelf dataset composed solely of favorable images. Experimental results on both synthetic images from the Google Scanned Objects dataset and real images from the OmniObject3D dataset validate the effectiveness of our method in handling unfavorable input views.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。