用文字和图像指令生成高保真动态3D世界，效率提升90倍
LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generation
- 结合轻量级大模型与UE5引擎，支持文本+图像多模态输入
- 生成场景布局准确度高，物理模拟真实，渲染实时流畅
- 工业生产效率提升90倍，适合游戏/自动驾驶等复杂场景开发
近年来，研究越来越关注构建能模拟复杂现实场景的3D世界模型，广泛应用于具身智能、自动驾驶、娱乐等领域。更真实的仿真能有效缩小仿真到现实的差距，便于获取丰富的真实世界信息。传统手工建模虽可创建虚拟3D场景，但现代方法正利用先进机器学习算法实现3D世界自动生成，尤其聚焦于根据用户指令生成虚拟世界。本文提出LatticeWorld，一个简单高效的3D世界生成框架，旨在简化3D环境的工业生产流程。该框架结合轻量级大语言模型（LLaMA-2-7B）与行业级渲染引擎（如Unreal Engine 5），接收文本描述和视觉指令作为多模态输入，生成包含动态代理的大规模3D交互世界，具备优异的多智能体互动能力、高保真物理模拟和实时渲染性能。我们进行了全面实验评估，结果表明其在场景布局生成准确性和视觉保真度方面表现卓越。相比传统人工制作方式，LatticeWorld实现了超过90倍的生产效率提升，同时保持高水平的创意质量。演示视频见：https://youtu.be/8VWZXpERR18
原文摘要 · Abstract (English)
Recent research has been increasingly focusing on developing 3D world models that simulate complex real-world scenarios. World models have found broad applications across various domains, including embodied AI, autonomous driving, entertainment, etc. A more realistic simulation with accurate physics will effectively narrow the sim-to-real gap and allow us to gather rich information about the real world conveniently. While traditional manual modeling has enabled the creation of virtual 3D scenes, modern approaches have leveraged advanced machine learning algorithms for 3D world generation, with most recent advances focusing on generative methods that can create virtual worlds based on user instructions. This work explores such a research direction by proposing LatticeWorld, a simple yet effective 3D world generation framework that streamlines the industrial production pipeline of 3D environments. LatticeWorld leverages lightweight LLMs (LLaMA-2-7B) alongside the industry-grade rendering engine (e.g., Unreal Engine 5) to generate a dynamic environment. Our proposed framework accepts textual descriptions and visual instructions as multimodal inputs and creates large-scale 3D interactive worlds with dynamic agents, featuring competitive multi-agent interaction, high-fidelity physics simulation, and real-time rendering. We conduct comprehensive experiments to evaluate LatticeWorld, showing that it achieves superior accuracy in scene layout generation and visual fidelity. Moreover, LatticeWorld achieves over a $90\times$ increase in industrial production efficiency while maintaining high creative quality compared with traditional manual production methods. Our demo video is available at https://youtu.be/8VWZXpERR18
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。