用2D数据做3D生成的隐式几何约束,统一文本到2D/3D生成。
Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation
- 将文本、图像、3D统一为离散序列,用2D数据指导3D建模。
- 在无完整三元组数据下仍实现文本引导3D生成与编辑性能提升。
- 适合研究多模态3D生成、缺乏高质量3D数据的团队使用。
近期多模态大模型在文本与图像理解生成方面表现优异,但将其能力扩展至3D仍受制于数据稀缺。相比丰富的2D图像,高质量3D资产稀少,导致3D生成任务约束不足。现有方法常依赖间接流程:在2D编辑后通过优化提升至3D,牺牲几何一致性。本文提出Omni123,一个3D原生基础模型,将文本到2D与文本到3D生成统一于单一自回归框架。核心思想是利用图像与3D之间的跨模态一致性作为隐式结构约束。通过将文本、图像和3D表示为共享序列空间中的离散标记,模型借助丰富的2D数据作为几何先验以改进3D表示。我们引入交错的X-to-X训练范式,在异构配对数据集上协调多样跨模态任务,无需完全对齐的文本-图像-3D三元组。通过在自回归序列中遍历语义-视觉-几何循环(如文本→图像→3D→图像),模型联合强化语义对齐、外观保真度及多视角几何一致性。实验表明,Omni123显著提升了文本引导的3D生成与编辑性能,为构建多模态3D世界模型提供了可扩展路径。
原文摘要 · Abstract (English)
Recent multimodal large language models have achieved strong performance in unified text and image understanding and generation, yet extending such native capability to 3D remains challenging due to limited data. Compared to abundant 2D imagery, high-quality 3D assets are scarce, making 3D synthesis under-constrained. Existing methods often rely on indirect pipelines that edit in 2D and lift results into 3D via optimization, sacrificing geometric consistency. We present Omni123, a 3D-native foundation model that unifies text-to-2D and text-to-3D generation within a single autoregressive framework. Our key insight is that cross-modal consistency between images and 3D can serve as an implicit structural constraint. By representing text, images, and 3D as discrete tokens in a shared sequence space, the model leverages abundant 2D data as a geometric prior to improve 3D representations. We introduce an interleaved X-to-X training paradigm that coordinates diverse cross-modal tasks over heterogeneous paired datasets without requiring fully aligned text-image-3D triplets. By traversing semantic-visual-geometric cycles (e.g., text to image to 3D to image) within autoregressive sequences, the model jointly enforces semantic alignment, appearance fidelity, and multi-view geometric consistency. Experiments show that Omni123 significantly improves text-guided 3D generation and editing, demonstrating a scalable path toward multimodal 3D world models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。