让大模型直接用文本生成3D网格，打通语言与空间创作的桥梁。
LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models
- 将3D网格顶点和面信息转为纯文本，让大模型无缝处理
- 能根据文字提示生成网格，还能交替输出文本与网格
- 首次实现大模型通过微调掌握复杂空间知识，适合创意设计者
本工作探索将预训练于文本的大语言模型（LLMs）能力拓展至统一生成3D网格。其优势在于：（1）利用文本来源如3D教程中已嵌入的空间知识；（2）支持对话式3D生成与网格理解。核心挑战在于如何将3D网格数据有效分词为模型可处理的离散标记。为此，我们提出LLaMA-Mesh，将3D网格的顶点坐标与面定义表示为纯文本，无需扩展词汇表即可直接接入大模型。构建了监督微调（SFT）数据集，使预训练大模型能够：（1）从文本提示生成3D网格；（2）按需生成交错的文本与3D网格输出；（3）理解并解释3D网格。本工作首次证明大模型可通过微调习得复杂空间知识以进行3D网格生成，实现文本与3D模态的统一。LLaMA-Mesh在网格生成质量上达到与从零训练模型相当的水平，同时保持强文本生成性能。
原文摘要 · Abstract (English)
This work explores expanding the capabilities of large language models (LLMs) pretrained on text to generate 3D meshes within a unified model. This offers key advantages of (1) leveraging spatial knowledge already embedded in LLMs, derived from textual sources like 3D tutorials, and (2) enabling conversational 3D generation and mesh understanding. A primary challenge is effectively tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly. To address this, we introduce LLaMA-Mesh, a novel approach that represents the vertex coordinates and face definitions of 3D meshes as plain text, allowing direct integration with LLMs without expanding the vocabulary. We construct a supervised fine-tuning (SFT) dataset enabling pretrained LLMs to (1) generate 3D meshes from text prompts, (2) produce interleaved text and 3D mesh outputs as required, and (3) understand and interpret 3D meshes. Our work is the first to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge for 3D mesh generation in a text-based format, effectively unifying the 3D and text modalities. LLaMA-Mesh achieves mesh generation quality on par with models trained from scratch while maintaining strong text generation performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。