arXiv:2410.05097cs.CVcs.LG2024-10被引 9

用单张图生成高精度航天器3D模型,支持太空目标新视角合成。

DreamSat: Towards a General 3D Model for Novel View Synthesis of Space Objects

  • 基于零样本3D重建模型微调,融合扩散与3D高斯泼溅技术。
  • 在30张未见航天器图像上,各项指标提升超2%,细节更真实。
  • 适合太空态势感知、交会对接等场景,无需为每类目标重训练。

新视角合成(NVS)可将一组2D图像转换为完整3D模型,在太空态势感知中对空间物体与碎片进行精准建模,提升太空操作的安全性与效率。在交会与近距离操作任务中,3D模型能提供目标的形状、尺寸与姿态信息,有助于行为预测与任务规划。本文提出DreamSat,一种从单视图图像重建通用3D航天器模型的新方法。通过在190个高质量航天器模型数据集上微调当前领先的单视图重建模型Zero123 XL,并集成至DreamGaussian框架,实现高效高精度重建。在包含30张未见过的航天器图像的测试集上,模型在多个指标上均取得显著提升:CLIP得分提升0.33%,峰值信噪比(PSNR)提高2.53%,结构相似性指数(SSIM)提升2.38%,学习感知图像块相似性(LPIPS)下降0.16%。该方法填补了航天领域专用3D重建工具的空白,结合先进的扩散模型与3D高斯泼溅技术,在保持高效的同时大幅提升重建精度。代码已开源:https://github.com/ARCLab-MIT/space-nvs。

原文摘要 · Abstract (English)

Novel view synthesis (NVS) enables to generate new images of a scene or convert a set of 2D images into a comprehensive 3D model. In the context of Space Domain Awareness, since space is becoming increasingly congested, NVS can accurately map space objects and debris, improving the safety and efficiency of space operations. Similarly, in Rendezvous and Proximity Operations missions, 3D models can provide details about a target object's shape, size, and orientation, allowing for better planning and prediction of the target's behavior. In this work, we explore the generalization abilities of these reconstruction techniques, aiming to avoid the necessity of retraining for each new scene, by presenting a novel approach to 3D spacecraft reconstruction from single-view images, DreamSat, by fine-tuning the Zero123 XL, a state-of-the-art single-view reconstruction model, on a high-quality dataset of 190 high-quality spacecraft models and integrating it into the DreamGaussian framework. We demonstrate consistent improvements in reconstruction quality across multiple metrics, including Contrastive Language-Image Pretraining (CLIP) score (+0.33%), Peak Signal-to-Noise Ratio (PSNR) (+2.53%), Structural Similarity Index (SSIM) (+2.38%), and Learned Perceptual Image Patch Similarity (LPIPS) (+0.16%) on a test set of 30 previously unseen spacecraft images. Our method addresses the lack of domain-specific 3D reconstruction tools in the space industry by leveraging state-of-the-art diffusion models and 3D Gaussian splatting techniques. This approach maintains the efficiency of the DreamGaussian framework while enhancing the accuracy and detail of spacecraft reconstructions. The code for this work can be accessed on GitHub (https://github.com/ARCLab-MIT/space-nvs).

3D重建航天器建模扩散模型新视角合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。