让大模型原生理解生成3D网格,支持精准编辑。
EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers

- 用混合变换器架构分离理解与生成专家,共享全局注意力
- 文本到3D生成质量达当前最优,支持长对话多轮几何编辑
- 适合需要3D原生能力的AI设计、游戏开发场景
本文解决将3D网格作为原生模态融入多模态大语言模型(MLLM)的挑战。基于扩散的重建模型将语义理解与几何推理分离,仅依赖密集2D像素先验进行状态无关重建。现有基于MLLM的方法将3D视为外部输出而非多模态序列的原生部分,缺乏对几何流形与MLLM特征空间对齐的系统分析。我们提出EVA01,一个统一框架,扩展了MLLM的模态边界,实现3D网格的原生理解、生成与上下文感知编辑。基于混合变换器(MoT)架构,EVA01将模型分解为预训练的理解专家(E_und)与结构镜像的生成专家(E_gen),通过共享全局自注意力与硬模态路由连接。该设计使MLLM主干的语义潜在空间与几何流形对齐,实现多模态先验的直接传递,无需中间2D表示。结果表明,EVA01在原生文本到3D生成保真度上达到当前最优,并实现了具有身份保持能力的鲁棒长上下文多轮几何编辑,这是无状态重建流水线无法实现的能力。研究进一步为2D基础模型与3D任务融合提供架构洞见,指导3D原生多模态系统的设计。
原文摘要 · Abstract (English)
This paper addresses the challenge of integrating 3D meshes as a native modality within Multimodal Large Language Models (MLLMs). Diffusion-based large reconstruction models decouple semantic understanding from geometric reasoning, operating as stateless reconstructors conditioned on dense 2D pixel priors. Recent MLLM-based methods treat the 3D modality as an external output rather than a native component of the multimodal sequence, making incremental adaptations without a systematic analysis of how geometric manifolds align with MLLM feature spaces. We introduce EVA01, a unified framework that extends the modality boundary of MLLMs to natively incorporate 3D mesh understanding, generation, and context-aware editing. Built upon a Mixture-of-Transformers (MoT) architecture, EVA01 decouples the model into a pre-trained Understanding Expert ($E_{\mathrm{und}}$) and a structurally mirrored Generation Expert ($E_{\mathrm{gen}}$), coupled through shared global self-attention with hard modality routing. This design aligns the semantic latent space of the MLLM backbone with the geometric manifold, enabling direct transfer of multimodal priors without intermediate 2D representations. Results show that EVA01 achieves state-of-the-art native text-to-3D generation fidelity and unlocks robust long-context multi-turn geometric editing with identity preservation, a capability fundamentally inaccessible to stateless reconstruction pipelines. Our findings further offer architectural insights for integrating 2D foundation models with 3D tasks, informing the design of 3D-native multimodal systems. Project Page: https://www.seeles.ai/research/pages/EVA01
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。