用文字描述生成多样3D场景,还能按用户反馈修改。
HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editing
- 结合视觉语言模型解析文本,生成高保真3D资产并布局。
- 支持室内与开放域场景,生成结果与描述高度一致。
- 可交互编辑布局和风格,适合游戏、虚拟现实等应用。
3D场景生成在游戏、艺术创作、虚拟现实等领域至关重要。然而,当前3D设计仍依赖大量人工,现有自动化方法难以生成开放域场景或支持灵活编辑。为此,我们提出HOLODECK 2.0,一个基于视觉-语言模型的3D世界生成框架,支持基于人类反馈的交互式场景编辑。该框架可生成多样且风格丰富的3D场景(如写实、卡通、动漫、赛博朋克),具备对细粒度文本描述的高度语义保真性,适用于室内及开放域环境。它利用视觉语言模型(VLMs)识别并解析场景所需对象,并通过先进的3D生成模型生成高质量资产;再通过迭代应用由VLMs导出的空间约束,实现语义一致且物理合理的布局。人评与模型评估均表明,HOLODECK 2.0在室内与开放域场景中均显著优于基线,生成效果与详细文本描述高度对齐。此外,该框架支持灵活编辑,可适应用户反馈进行布局优化与风格一致的对象修改。最后,我们在程序化游戏建模中展示了其实际应用,成功生成视觉丰富、沉浸感强的环境,大幅提升游戏设计效率。代码已开源:https://github.com/bzx20/Holodeck2.0。
原文摘要 · Abstract (English)
3D scene generation plays a crucial role in gaming, artistic creation, virtual reality, and many other domains. However, current 3D scene design still relies heavily on extensive manual effort from creators, and existing automated methods struggle to generate open-domain scenes or support flexible editing. To address those challenges, we introduce HOLODECK 2.0, an advanced vision-language-guided framework for 3D world generation with support for interactive scene editing based on human feedback. HOLODECK 2.0 can generate diverse and stylistically rich 3D scenes (e.g., realistic, cartoon, anime, and cyberpunk styles) that exhibit high semantic fidelity to fine-grained input descriptions, suitable for both indoor and open-domain environments. HOLODECK 2.0 leverages vision-language models (VLMs) to identify and parse the objects required in a scene and generates corresponding high-quality assets via state-of-the-art 3D generative models. Then, HOLODECK 2.0 iteratively applies spatial constraints derived from the VLMs to achieve semantically coherent and physically plausible layouts. Both human and model evaluations demonstrate that HOLODECK 2.0 effectively generates high-quality scenes closely aligned with detailed textual descriptions, consistently outperforming baselines across indoor and open-domain scenarios. Additionally, HOLODECK 2.0 provides editing capabilities that flexibly adapt to human feedback, supporting layout refinement and style-consistent object edits. Finally, we present a practical application of HOLODECK 2.0 in procedural game modeling to generate visually rich and immersive environments that can boost efficiency in game design Code is available at https://github.com/bzx20/Holodeck2.0.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。