用几何图结构生成3D室内场景,无需预设关系标签
GeoSceneGraph: Geometric Scene Graph Diffusion Model for Text-guided 3D Indoor Scene Synthesis
- 基于等变图神经网络,利用物体间几何对称性建模场景结构
- 在无真实关系标注下达到与依赖标注方法相当的生成效果
- 适合资源受限设备,如XR眼镜、手机端部署
从文本提示生成3D室内场景的方法在影视制作、室内设计、游戏开发、虚拟现实及训练具身智能体的合成数据中具有广泛应用。现有方法通常从头训练生成模型或利用视觉语言模型(VLMs)。尽管VLMs在复杂或开放性提示下表现优异,但小型专用模型仍需用于资源受限设备(如扩展现实眼镜、手机)。然而,许多从头训练的方法忽略了室内场景固有的图结构,影响场景一致性和真实性。而采用场景图的方法要么需要用户提供的语义图(不便且限制性强),要么依赖真实关系标注,难以捕捉多样化的物体交互。为此,我们提出GeoSceneGraph,一种通过利用3D场景的图结构和几何对称性,从文本提示生成3D场景的方法,无需预定义关系类别。尽管未使用真实关系标注,其性能仍可媲美依赖标注的方法。模型基于等变图神经网络(EGNNs),但现有EGNN方法通常仅支持低维条件输入,不适用于文本等复杂模态。我们提出一种简单有效的策略,将文本特征融入EGNN,通过消融实验验证设计有效性。
原文摘要 · Abstract (English)
Methods that synthesize indoor 3D scenes from text prompts have wide-ranging applications in film production, interior design, video games, virtual reality, and synthetic data generation for training embodied agents. Existing approaches typically either train generative models from scratch or leverage vision-language models (VLMs). While VLMs achieve strong performance, particularly for complex or open-ended prompts, smaller task-specific models remain necessary for deployment on resource-constrained devices such as extended reality (XR) glasses or mobile phones. However, many generative approaches that train from scratch overlook the inherent graph structure of indoor scenes, which can limit scene coherence and realism. Conversely, methods that incorporate scene graphs either demand a user-provided semantic graph, which is generally inconvenient and restrictive, or rely on ground-truth relationship annotations, limiting their capacity to capture more varied object interactions. To address these challenges, we introduce GeoSceneGraph, a method that synthesizes 3D scenes from text prompts by leveraging the graph structure and geometric symmetries of 3D scenes, without relying on predefined relationship classes. Despite not using ground-truth relationships, GeoSceneGraph achieves performance comparable to methods that do. Our model is built on equivariant graph neural networks (EGNNs), but existing EGNN approaches are typically limited to low-dimensional conditioning and are not designed to handle complex modalities such as text. We propose a simple and effective strategy for conditioning EGNNs on text features, and we validate our design through ablation studies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。