用现成3D重建模型当编码器,高效生成高质量文本驱动3D内容。
Taming Feed-forward Reconstruction Models as Latent Encoders for 3D Generative Models
- 复用预训练重建模型提取3D特征,省去昂贵编码器训练
- 通过空间加权和感知损失,使不规则特征适合生成建模
- 多流Transformer结构实现线性扩展,支持高质量文本生成3D
近期基于AI的3D内容生成主要沿两条路径发展:前馈式图像到3D重建方法,以及使用2D或3D监督训练的3D生成模型。本文证明,现有前馈重建方法可作为3D生成模型的有效潜在编码器,从而融合这两种范式。通过复用强大的预训练重建模型,我们避免了计算成本高昂的编码器训练,并免费获得丰富的3D潜在特征。然而,重建模型的潜在空间因结构不规则,不适合生成建模。为此,我们开发了后处理流程,包括特征标准化协议和空间加权机制,以聚焦重要区域。我们进一步引入2D图像空间感知渲染损失,以应对高维潜在空间。最终,提出一种基于多流Transformer的修正流架构,实现线性缩放与高质量文本条件3D生成。该框架利用前馈重建模型的进展,提升3D生成建模的可扩展性,在计算效率与文本到3D生成性能上均达当前最优水平。
原文摘要 · Abstract (English)
Recent AI-based 3D content creation has largely evolved along two paths: feed-forward image-to-3D reconstruction approaches and 3D generative models trained with 2D or 3D supervision. In this work, we show that existing feed-forward reconstruction methods can serve as effective latent encoders for training 3D generative models, thereby bridging these two paradigms. By reusing powerful pre-trained reconstruction models, we avoid computationally expensive encoder network training and obtain rich 3D latent features for generative modeling for free. However, the latent spaces of reconstruction models are not well-suited for generative modeling due to their unstructured nature. To enable flow-based model training on these latent features, we develop post-processing pipelines, including protocols to standardize the features and spatial weighting to concentrate on important regions. We further incorporate a 2D image space perceptual rendering loss to handle the high-dimensional latent spaces. Finally, we propose a multi-stream transformer-based rectified flow architecture to achieve linear scaling and high-quality text-conditioned 3D generation. Our framework leverages the advancements of feed-forward reconstruction models to enhance the scalability of 3D generative modeling, achieving both high computational efficiency and state-of-the-art performance in text-to-3D generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。