用自然语言生成可交互编辑的3D房间网格,支持精细控制布局与纹理。
Programmable-Room: Interactive Textured 3D Room Meshes Generation Empowered by Large Language Models
- 通过视觉编程框架,将自然语言指令转化为可执行的3D建模流程
- 利用扩散模型结合布局、深度和语义图生成高质量全景纹理图像
- 支持房间属性的细粒度编辑,适合场景设计与虚拟空间创作
我们提出Programmable-Room,一个基于自然语言指令交互生成与编辑3D房间网格的框架。为实现对房间各属性的精确控制,将复杂任务分解为创建合理3D坐标、生成全景纹理图、集成坐标与纹理构建网格、布置家具等步骤。为统一支持这些分解任务,引入视觉编程(VP),即利用大语言模型(LLM)将自然语言指令转换为类似Python的程序,包含一系列必要模块。我们开发了大部分模块,其中纹理生成模块采用预训练的大规模扩散模型,同时以文本和视觉提示(如布局、深度图、语义图)作为条件生成全景图像。特别地,通过双向LSTM提取全景场景的一维表示,优化训练目标以提升图像生成质量。实验表明,Programmable-Room在生成与编辑3D房间网格方面具有高度灵活性,并在定量与定性上优于现有模型。项目主页见https://jihyun0510.github.io/Programmable_Room_Page/
原文摘要 · Abstract (English)
We present Programmable-Room, a framework which interactively generates and edits a 3D room mesh, given natural language instructions. For precise control of a room's each attribute, we decompose the challenging task into simpler steps such as creating plausible 3D coordinates for room meshes, generating panorama images for the texture, constructing 3D meshes by integrating the coordinates and panorama texture images, and arranging furniture. To support the various decomposed tasks with a unified framework, we incorporate visual programming (VP). VP is a method that utilizes a large language model (LLM) to write a Python-like program which is an ordered list of necessary modules for the various tasks given in natural language. We develop most of the modules. Especially, for the texture generating module, we utilize a pretrained large-scale diffusion model to generate panorama images conditioned on text and visual prompts (i.e., layout, depth, and semantic map) simultaneously. Specifically, we enhance the panorama image generation quality by optimizing the training objective with a 1D representation of a panorama scene obtained from bidirectional LSTM. We demonstrate Programmable-Room's flexibility in generating and editing 3D room meshes, and prove our framework's superiority to an existing model quantitatively and qualitatively. Project page is available in https://jihyun0510.github.io/Programmable_Room_Page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。