用户输入文本或图像即可生成精细3D场景,无需懂设计图谱。
FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts
- 用视觉语言模型将文字/图像转为场景图谱,自动理解用户意图。
- 单模型支持文生图、图生图和场景重排,生成质量优于现有方法。
- 适合想快速生成个性化3D场景的设计师、开发者与创作者。
可控性在3D室内场景合成的实际应用中至关重要。现有方法要么仅支持粗粒度的文本控制(便捷但定制性差),要么依赖图结构控制(可控性强但需复杂图谱设计)。为此,我们提出FreeScene,一种用户友好的框架,实现便捷且高效的3D场景合成控制。FreeScene支持自由形式的输入,包括文本描述和/或参考图像,用户可灵活表达设计意图。这些输入由基于视觉语言模型的图谱设计模块分析并转化为图表示。随后,我们提出混合图谱扩散变换器MG-DiT,通过图感知去噪提升生成效果。MG-DiT不仅能有效保留图谱结构,还可泛化至多种任务,包括但不限于文生图、图生图及场景重排,均在单一模型中完成。大量实验表明,FreeScene在生成质量和可控性方面均超越当前最优方法,为文本与图谱驱动的场景合成提供统一高效解决方案。
原文摘要 · Abstract (English)
Controllability plays a crucial role in the practical applications of 3D indoor scene synthesis. Existing works either allow rough language-based control, that is convenient but lacks fine-grained scene customization, or employ graph based control, which offers better controllability but demands considerable knowledge for the cumbersome graph design process. To address these challenges, we present FreeScene, a user-friendly framework that enables both convenient and effective control for indoor scene synthesis.Specifically, FreeScene supports free-form user inputs including text description and/or reference images, allowing users to express versatile design intentions. The user inputs are adequately analyzed and integrated into a graph representation by a VLM-based Graph Designer. We then propose MG-DiT, a Mixed Graph Diffusion Transformer, which performs graph-aware denoising to enhance scene generation. Our MG-DiT not only excels at preserving graph structure but also offers broad applicability to various tasks, including, but not limited to, text-to-scene, graph-to-scene, and rearrangement, all within a single model. Extensive experiments demonstrate that FreeScene provides an efficient and user-friendly solution that unifies text-based and graph based scene synthesis, outperforming state-of-the-art methods in terms of both generation quality and controllability in a range of applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。