让大模型逐步理解生成3D网格，提升结构还原能力
MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Mesh
- 用基础网格分解法将3D网格拆成有意义的子单元
- 构建超大规模数据集（150万+样本），是之前50倍
- 适合想用大模型处理3D文本序列的研究者
我们提出MeshLLM，一种利用大语言模型（LLMs）理解与生成文本序列化3D网格的新框架。针对现有方法在适配LLM词元长度时数据规模有限、网格序列化导致3D结构信息丢失的问题，我们引入基础网格分解策略，将3D网格划分为具有结构意义的子单元。该方法促成构建一个包含150万+样本的大规模数据集，接近此前方法的50倍，更契合LLM扩展规律。此外，我们提出从顶点推断面连接关系及局部网格组装训练策略，显著增强大模型对网格拓扑和空间结构的捕捉能力。实验表明，MeshLLM在网格生成质量与形状理解上均优于当前最优的LLaMA-Mesh，展现出处理文本序列化3D网格的巨大潜力。
原文摘要 · Abstract (English)
We present MeshLLM, a novel framework that leverages large language models (LLMs) to understand and generate text-serialized 3D meshes. Our approach addresses key limitations in existing methods, including the limited dataset scale when catering to LLMs' token length and the loss of 3D structural information during mesh serialization. We introduce a Primitive-Mesh decomposition strategy, which divides 3D meshes into structurally meaningful subunits. This enables the creation of a large-scale dataset with 1500k+ samples, almost 50 times larger than previous methods, which aligns better with the LLM scaling law principles. Furthermore, we propose inferring face connectivity from vertices and local mesh assembly training strategies, significantly enhancing the LLMs' ability to capture mesh topology and spatial structures. Experiments show that MeshLLM outperforms the state-of-the-art LLaMA-Mesh in both mesh generation quality and shape understanding, highlighting its great potential in processing text-serialized 3D meshes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。