用大模型同时实现3D物体描述与高精度生成，突破现有方法局限。
CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models
- 分层建模：令牌级与块级自回归架构协同处理文本与空间结构
- 生成高保真3D模型，支持长上下文交互与精细几何细节
- 不仅生成3D内容，还提升图像理解能力，适合多模态应用
大型语言模型（LLMs）在文本生成与多模态感知方面已取得突破，但在3D内容生成方面仍待探索。现有方法通常生成低分辨率网格或粗糙结构代理，难以原生捕捉精细几何特征。本文提出CG-MLLM，一种新型多模态大语言模型，可在单一框架内完成3D描述与高分辨率3D生成。通过混合Transformer架构，模型分离不同建模需求：令牌级自回归（TokenAR）Transformer处理细粒度内容，块级自回归（BlockAR）Transformer处理块级结构。结合预训练视觉-语言主干与专用3D VAE隐空间，实现标准令牌与空间块间的长上下文交互。实验表明，CG-MLLM在生成高保真3D对象方面显著优于现有多模态大模型，将高分辨率3D内容生成引入主流大模型范式。此外，学习生成3D内容还反向增强模型的图像基础3D理解能力。
原文摘要 · Abstract (English)
Large Language Models(LLMs) have revolutionized text generation and multimodal perception,but their capabilities in 3D content generation remain underexplored. Existing methods compromise by producing either low-resolution meshes or coarse structural proxies, failing to capture finegrained geometry natively. In this paper, we propose CG-MLLM, a novel Multi-modal Large Language Model (MLLM) capable of 3D captioning and high-resolution 3D generation in a single framework. Leveraging the Mixture-ofTransformer architecture, CG-MLLM decouples disparate modeling needs, where the Token-level Autoregressive (TokenAR) Transformer handles token-level content, and the Block-level Autoregressive (BlockAR) Transformer handles blocklevel content. By integrating a pre-trained visionlanguage backbone with a specialized 3D VAE latent space, CG-MLLM facilitates long-context interactions between standard tokens and spatial blocks within a single integrated architecture. Experimental results show that CG-MLLM significantly outperforms existing MLLMs in generating high-fidelity 3D objects, effectively bringing high-resolution 3D content creation into the mainstream LLM paradigm. Beyond generation, we further observe that learning to produce 3D content transfers back to perception, strengthening the model's image-based 3D understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。