用自校准方法让3D模型快速适应新场景,5分钟搞定无需人工标注。
LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Models
- 基于多视角预测自动生成伪标签,通过低秩适配微调模型。
- 在160多个场景上实现最高88%的性能提升,单卡5分钟完成。
- 无需外部先验或人工标注,仅需18MB存储空间,适合部署到边缘设备。
新兴的3D几何基础模型(如DUSt3R)为真实世界中的3D视觉任务提供了前景。然而,由于问题空间的高维性和高质量3D数据的稀缺,这些预训练模型在视图重叠少或光照不足等挑战性场景中仍难以泛化。为此,我们提出LoRA3D,一种高效的自校准流程,利用模型自身的多视角预测将预训练模型特化到目标场景。输入稀疏的RGB图像后,采用鲁棒优化技术精炼多视角预测并统一至全局坐标系。特别地,将预测置信度融入几何优化过程,自动重加权以更准确反映点估计精度。利用校准后的置信度生成高质量伪标签,并使用低秩适配(LoRA)在伪标签数据上微调模型。本方法不依赖任何外部先验或人工标注,在单张标准GPU上仅需5分钟完成自校准,每个低秩适配器仅需18MB存储空间。我们在Replica、TUM和Waymo Open数据集超过160个场景上进行了评估,3D重建、多视图位姿估计和新视角渲染任务上最高提升达88%。
原文摘要 · Abstract (English)
Emerging 3D geometric foundation models, such as DUSt3R, offer a promising approach for in-the-wild 3D vision tasks. However, due to the high-dimensional nature of the problem space and scarcity of high-quality 3D data, these pre-trained models still struggle to generalize to many challenging circumstances, such as limited view overlap or low lighting. To address this, we propose LoRA3D, an efficient self-calibration pipeline to $\textit{specialize}$ the pre-trained models to target scenes using their own multi-view predictions. Taking sparse RGB images as input, we leverage robust optimization techniques to refine multi-view predictions and align them into a global coordinate frame. In particular, we incorporate prediction confidence into the geometric optimization process, automatically re-weighting the confidence to better reflect point estimation accuracy. We use the calibrated confidence to generate high-quality pseudo labels for the calibrating views and use low-rank adaptation (LoRA) to fine-tune the models on the pseudo-labeled data. Our method does not require any external priors or manual labels. It completes the self-calibration process on a $\textbf{single standard GPU within just 5 minutes}$. Each low-rank adapter requires only $\textbf{18MB}$ of storage. We evaluated our method on $\textbf{more than 160 scenes}$ from the Replica, TUM and Waymo Open datasets, achieving up to $\textbf{88% performance improvement}$ on 3D reconstruction, multi-view pose estimation and novel-view rendering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。