用知识蒸馏让小模型复现大模型的3D重建效果
Multi-View 3D Reconstruction using Knowledge Distillation
- 用Dust3r做教师，训练轻量学生模型学习3D点云
- Vision Transformer架构在视觉和定量指标上表现最佳
- 适合需要高效3D重建的视觉定位场景
大型基础模型如Dust3r可基于立体图像对生成高质量点云、相机内参和深度估计，但其在视觉定位等任务中推理耗时长、资源消耗大。本文提出一种知识蒸馏框架，以Dust3r为教师模型，训练多种学生模型，利用其输出的3D点云进行学习。目标是构建能捕捉场景特性的轻量级学生模型，实现与Dust3r相当的3D重建性能。实验使用12Scenes数据集，对比了基于CNN和基于视觉变压器（Vision Transformer）的学生模型，并考察了预训练与从零开始训练的差异。定性分析显示学生模型输出的3D点云与Dust3r接近，且具备学习特定场景特征的能力。通过超参数调优进行消融研究，结果表明视觉变压器架构在视觉质量和定量评估上均表现最优。
原文摘要 · Abstract (English)
Large Foundation Models like Dust3r can produce high quality outputs such as pointmaps, camera intrinsics, and depth estimation, given stereo-image pairs as input. However, the application of these outputs on tasks like Visual Localization requires a large amount of inference time and compute resources. To address these limitations, in this paper, we propose the use of a knowledge distillation pipeline, where we aim to build a student-teacher model with Dust3r as the teacher and explore multiple architectures of student models that are trained using the 3D reconstructed points output by Dust3r. Our goal is to build student models that can learn scene-specific representations and output 3D points with replicable performance such as Dust3r. The data set we used to train our models is 12Scenes. We test two main architectures of models: a CNN-based architecture and a Vision Transformer based architecture. For each architecture, we also compare the use of pre-trained models against models built from scratch. We qualitatively compare the reconstructed 3D points output by the student model against Dust3r's and discuss the various features learned by the student model. We also perform ablation studies on the models through hyperparameter tuning. Overall, we observe that the Vision Transformer presents the best performance visually and quantitatively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。