三秒内从图像生成完整3D形状,无需标定相机
Fus3D: Decoding Consolidated 3D Geometry from Feed-forward Geometry Transformer Latents
- 直接从预训练模型特征提取3D体素表示,避免逐视图预测误差累积
- 在稀疏和密集视图下均生成完整且连续的符号距离场
- 适合需要快速高质量3D重建的应用,如虚拟现实与机器人
我们提出一种前馈方法,仅需三秒即可从无序图像集合中回归密集符号距离场(SDF),无需相机标定或后处理融合。核心洞察是:预训练多视角前馈几何变换器的中间特征空间已编码强大联合世界表征;然而现有流程丢弃该信息,先通过单视图预测头生成结果,再后期拼接,导致完整性损失和误差累积。本文直接从几何变换器特征中进行3D提取,采用可学习的体素化提取机制:通过交替交叉注意力与自注意力,逐步将多视图几何信息融入结构化的体素潜在网格。一个简单的卷积解码器将该网格映射为密集SDF。此外,我们提出一种可扩展、有效性感知的监督方案,直接使用深度图或3D资产生成的SDF作为监督信号,解决非水密网格等实际问题。方法在稀疏与密集视图设置下均生成完整且定义明确的距离值,实现几何上合理的补全。代码与更多材料见 https://lorafib.github.io/fus3d。
原文摘要 · Abstract (English)
We propose a feed-forward method for dense Signed Distance Field (SDF) regression from unstructured image collections in less than three seconds, without camera calibration or post-hoc fusion. Our key insight is that the intermediate feature space of pretrained multi-view feed-forward geometry transformers already encodes a powerful joint world representation; yet, existing pipelines discard it, routing features through per-view prediction heads before assembling 3D geometry post-hoc, which discards valuable completeness information and accumulates inaccuracies. We instead perform 3D extraction directly from geometry transformer features via learned volumetric extraction: voxelized canonical embeddings that progressively absorb multi-view geometry information through interleaved cross- and self-attention into a structured volumetric latent grid. A simple convolutional decoder then maps this grid to a dense SDF. We additionally propose a scalable, validity-aware supervision scheme directly using SDFs derived from depth maps or 3D assets, tackling practical issues like non-watertight meshes. Our approach yields complete and well-defined distance values across sparse- and dense-view settings and demonstrates geometrically plausible completions. Code and further material can be found at https://lorafib.github.io/fus3d.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。