用神经网络生成可解析的3D程序图，从图片重建复杂3D模型。
ProcGen3D: Learning Neural Procedural Graph Representations for Image-to-3D Reconstruction
- 通过图结构编码程序化3D生成逻辑，实现可学习的3D内容表示。
- 在仙人掌、树木和桥梁上优于主流生成方法，真实图像泛化能力强。
- 结合蒙特卡洛树搜索优化生成路径，输出更贴合输入图像。
我们提出ProcGen3D，一种基于图像的3D重建新方法，通过生成3D物体的程序化图抽象来创建复杂3D资产。受生产环境中广泛使用程序化生成器的启发，我们设计了一种序列化的基于图的程序化图表示，并训练神经网络以逼近程序化生成器的潜在空间。采用基于边的标记化对程序化图进行编码，使用Transformer先验模型根据输入RGB图像预测下一个标记。关键在于，为提升生成结果与输入图像的对齐度，我们在生成过程中引入蒙特卡洛树搜索（MCTS）引导采样，使生成的程序化图更贴近输入图像。该方法适用于多种可通过程序化生成的物体。在仙人掌、树木和桥梁上的大量实验表明，我们的神经程序化图生成方法在性能上优于当前最先进的生成式3D方法及领域专用建模技术。此外，尽管仅在合成数据上训练，仍能在真实图像上实现良好泛化。
原文摘要 · Abstract (English)
We introduce ProcGen3D, a new approach for 3D content creation by generating procedural graph abstractions of 3D objects, which can then be decoded into rich, complex 3D assets. Inspired by the prevalent use of procedural generators in production 3D applications, we propose a sequentialized, graph-based procedural graph representation for 3D assets. We use this to learn to approximate the landscape of a procedural generator for image-based 3D reconstruction. We employ edge-based tokenization to encode the procedural graphs, and train a transformer prior to predict the next token conditioned on an input RGB image. Crucially, to enable better alignment of our generated outputs to an input image, we incorporate Monte Carlo Tree Search (MCTS) guided sampling into our generation process, steering output procedural graphs towards more image-faithful reconstructions. Our approach is applicable across a variety of objects that can be synthesized with procedural generators. Extensive experiments on cacti, trees, and bridges show that our neural procedural graph generation outperforms both state-of-the-art generative 3D methods and domain-specific modeling techniques. Furthermore, this enables improved generalization on real-world input images, despite training only on synthetic data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。