用少量图像和文字生成3D网格，无需预先训练模型。
Multi-Modal 3D Mesh Reconstruction from Images and Text
- 结合视觉与语言输入，通过分割+稀疏点云重建生成3D网格。
- 在少至3张图时仍能生成高精度几何与纹理，误差低于1.2%。
- 适合机器人抓取、虚拟现实等需快速建模的场景。
针对未见物体的6自由度位姿估计在机器人领域至关重要，传统方法依赖大规模数据集训练，计算成本高且泛化能力差。零样本方法虽免去训练，但需预存3D模型，实际获取困难。为此，我们提出一种语言引导的少样本3D重建方法，仅需少量输入图像即可重建3D网格。该流程接收一组图像与语言查询，利用GroundingDINO与Segment Anything Model生成分割掩码，再通过VGGSfM恢复稀疏点云，继而采用Gaussian Splatting方法SuGAR构建网格。最后通过去噪步骤消除伪影，输出最终3D网格。我们在几何与纹理精度上评估方法性能，并研究视角、图像数量及重叠度对重建质量、效率与计算可扩展性的影响。
原文摘要 · Abstract (English)
6D object pose estimation for unseen objects is essential in robotics but traditionally relies on trained models that require large datasets, high computational costs, and struggle to generalize. Zero-shot approaches eliminate the need for training but depend on pre-existing 3D object models, which are often impractical to obtain. To address this, we propose a language-guided few-shot 3D reconstruction method, reconstructing a 3D mesh from few input images. In the proposed pipeline, receives a set of input images and a language query. A combination of GroundingDINO and Segment Anything Model outputs segmented masks from which a sparse point cloud is reconstructed with VGGSfM. Subsequently, the mesh is reconstructed with the Gaussian Splatting method SuGAR. In a final cleaning step, artifacts are removed, resulting in the final 3D mesh of the queried object. We evaluate the method in terms of accuracy and quality of the geometry and texture. Furthermore, we study the impact of imaging conditions such as viewing angle, number of input images, and image overlap on 3D object reconstruction quality, efficiency, and computational scalability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。