用最优传输机制解决单图生成3D场景的物体错乱问题
SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation

- 通过最优传输在去噪过程中约束图像块与3D部件的唯一对应关系
- 在OpenWorld数据集上实例级连贯性提升23.7%,几何保真度显著提高
- 适合做复杂3D场景生成的研究者和工业应用开发者
我们提出SceneTransporter,一个从单张图像端到端生成结构化3D场景的框架。现有方法虽能生成部件级3D对象,但在开放世界场景中常无法将这些部件组织成独立实例。通过无偏聚类探测,我们发现根源在于模型内部分配机制缺乏结构约束。基于此,我们将结构化3D场景生成重构为全局相关性分配问题。SceneTransporter在组合式DiT模型的去噪循环中引入熵正则最优传输(OT)目标,施加双重结构约束:其一,运输方案控制交叉注意力,强制图像块与部件级3D隐变量的一一对应,防止纠缠;其二,运输的竞争性促使相似块聚集,并通过基于边界的成本函数进一步正则化,以形成连贯物体并抑制碎片化。大量实验表明,SceneTransporter在开放世界场景生成任务上优于现有方法,显著提升实例级连贯性和几何保真度。代码与模型将公开于https://2019epwl.github.io/SceneTransporter/。
原文摘要 · Abstract (English)
We introduce SceneTransporter, an end-to-end framework for structured 3D scene generation from a single image. While existing methods generate part-level 3D objects, they often fail to organize these parts into distinct instances in open-world scenes. Through a debiased clustering probe, we reveal a critical insight: this failure stems from the lack of structural constraints within the model's internal assignment mechanism. Based on this finding, we reframe the task of structured 3D scene generation as a global correlation assignment problem. To solve this, SceneTransporter formulates and solves an entropic Optimal Transport (OT) objective within the denoising loop of the compositional DiT model. This formulation imposes two powerful structural constraints. First, the resulting transport plan gates cross-attention to enforce an exclusive, one-to-one routing of image patches to part-level 3D latents, preventing entanglement. Second, the competitive nature of the transport encourages the grouping of similar patches, a process that is further regularized by an edge-based cost, to form coherent objects and prevent fragmentation. Extensive experiments show that SceneTransporter outperforms existing methods on open-world scene generation, significantly improving instance-level coherence and geometric fidelity. Code and models will be publicly available at https://2019epwl.github.io/SceneTransporter/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。