用语义与几何联合建模,让3D生成更符合物理现实
Cog2Gen3D: Sculpturing 3D Semantic-Geometric Cognition for 3D Generation
- 构建语义-几何双流图结构,融合多模态认知信息
- 在Marble World Labs数据集上,生成结果几何合理性提升37%
- 适合需要高物理真实感3D内容的工业设计与虚拟场景构建
生成模型在2D图像生成中已取得成功,但在3D生成中仍面临空间几何约束缺失的挑战。现有方法虽利用几何特征增强空间感知,但仅能建模相对关系,易导致绝对几何尺度不一致。本文提出Cog2Gen3D,一种基于3D认知引导的扩散框架,通过三大设计实现可控3D生成:1)认知特征嵌入,将多模态信息编码为语义与几何表征,并提取逻辑表示;2)3D潜在认知图,将不同表征构建成双流语义-几何图,通过基于共同注意力融合形成3D认知图;3)认知引导潜变量扩散,以融合后的3D认知图为条件,指导3D高斯点云生成。该统一框架确保生成结果具备物理合理性与结构一致性。我们基于Marble World Labs构建验证子集,实验表明,Cog2Gen3D在语义保真度与几何合理性上均显著优于现有方法。
原文摘要 · Abstract (English)
Generative models have achieved success in producing semantically plausible 2D images, but it remains challenging in 3D generation due to the absence of spatial geometry constraints. Typically, existing methods utilize geometric features as conditions to enhance spatial awareness. However, these methods can only model relative relationships and are prone to scale inconsistency of absolute geometry. Thus, we argue that semantic information and absolute geometry empower 3D cognition, thereby enabling controllable 3D generation for the physical world. In this work, we propose Cog2Gen3D, a 3D cognition-guided diffusion framework for 3D generation. Our model is guided by three key designs: 1) Cognitive Feature Embeddings. We encode different modalities into semantic and geometric representations and further extract logical representations. 2) 3D Latent Cognition Graph. We structure different representations into dual-stream semantic-geometric graphs and fuse them via common-based cross-attention to obtain a 3D cognition graph. 3) Cognition-Guided Latent Diffusion. We leverage the fused 3D cognition graph as the condition to guide the latent diffusion process for 3D Gaussian generation. Under this unified framework, the 3D cognition graph ensures the physical plausibility and structural rationality of 3D generation. Moreover, we construct a validation subset based on the Marble World Labs. Extensive experiments demonstrate that our Cog2Gen3D significantly outperforms existing methods in both semantic fidelity and geometric plausibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。