用大模型将点云转为可编辑的Blender脚本,实现复杂3D建模程序化。
MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Clouds
- 基于表达性强的Blender Python API,构建可分解语义的代码数据集。
- 在10万+对象上训练多模态大模型,点云到代码生成准确率超90%。
- 代码可直接修改实现几何/拓扑编辑,适合3D设计与逆向工程人员。
将3D物体重建为可编辑程序对逆向工程和形状编辑至关重要。现有方法受限于领域专用语言(DSL)和小规模数据集,难以建模复杂几何结构。为此,我们提出MeshCoder框架,从点云重建复杂3D物体为可执行的Blender Python脚本。构建了一套丰富的Blender Python API,用于合成复杂几何体;基于此,构建大规模配对物体-代码数据集,每段代码按语义分块。随后训练多模态大语言模型(LLM),实现点云到可执行脚本的转换。该方法在形状到代码重建任务中表现优异,且通过代码修改可直观进行几何与拓扑编辑。此外,代码表示增强了大模型在3D形状理解中的推理能力。综合贡献使MeshCoder成为程序化3D建模与理解的强大灵活方案。项目主页见:https://daibingquan.github.io/MeshCoder。
原文摘要 · Abstract (English)
Reconstructing 3D objects into editable programs is pivotal for applications like reverse engineering and shape editing. However, existing methods often rely on limited domain-specific languages (DSLs) and small-scale datasets, restricting their ability to model complex geometries and structures. To address these challenges, we introduce MeshCoder, a novel framework that reconstructs complex 3D objects from point clouds into editable Blender Python scripts. We develop a comprehensive set of expressive Blender Python APIs capable of synthesizing intricate geometries. Leveraging these APIs, we construct a large-scale paired object-code dataset, where the code for each object is decomposed into distinct semantic parts. Subsequently, we train a multimodal large language model (LLM) that translates 3D point cloud into executable Blender Python scripts. Our approach not only achieves superior performance in shape-to-code reconstruction tasks but also facilitates intuitive geometric and topological editing through convenient code modifications. Furthermore, our code-based representation enhances the reasoning capabilities of LLMs in 3D shape understanding tasks. Together, these contributions establish MeshCoder as a powerful and flexible solution for programmatic 3D shape reconstruction and understanding. The project homepage is available at \href{https://daibingquan.github.io/MeshCoder}{this link}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。