用自然语言生成和编辑3D室内场景,支持物体增删换且保持空间合理。
ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing
- 基于自回归文本驱动,显式建模房间边界与物体语义。
- 在物体添加任务上超越现有方法,合成场景人眼质量更优。
- 适合需要自然语言交互的3D场景生成与编辑应用。
场景生成与编辑已成为计算机图形学中的前沿方向。现有3D室内场景生成方法或使用单一类别编码简化语义(如‘椅子’或‘桌子’),或需掩码扩散模型进行编辑,忽略房间边界,或依赖平面图渲染而无法捕捉复杂布局。基于大语言模型的方法虽能通过自然语言实现丰富语义,但缺乏编辑功能,仅限矩形布局,或依赖隐式世界模型的弱空间推理。我们提出ReSpace,一个自回归、文本驱动的3D室内场景生成与编辑框架。该方法采用紧凑的结构化场景表示,显式建模房间边界,实现无资产依赖部署,并将场景操作转化为下一词预测任务,支持通过自然语言实现物体的添加、删除与替换。我们采用监督微调结合偏好对齐训练专用语言模型,以兼顾用户指令、空间几何、物体语义与整体构图。此外,引入基于体素的评估指标,捕捉超出3D包围盒的精细几何违规。实验表明,在物体添加任务上优于现有最佳方法,并在全场景生成应用中获得更优的人类感知质量,尽管未在该任务上进行训练。
原文摘要 · Abstract (English)
Scene synthesis and editing has emerged as a promising direction in computer graphics. Current trained approaches for 3D indoor scene generation either oversimplify object semantics through one-hot class encodings (e.g., 'chair' or 'table'), require masked diffusion for editing, ignore room boundaries, or rely on floor plan renderings that fail to capture complex layouts. LLM-based methods enable richer semantics via natural language, but lack editing functionality, are limited to rectangular layouts, or rely on weak spatial reasoning from implicit world models. We introduce ReSpace, a generative framework for autoregressive text-driven 3D indoor scene synthesis and editing. Our approach features a compact structured scene representation with explicit room boundaries that enables asset-agnostic deployment and frames scene manipulation as a next-token prediction task, supporting object addition, removal, and swapping via natural language. We employ supervised fine-tuning with a preference alignment stage to train a specialized language model for object addition that accounts for user instructions, spatial geometry, object semantics, and scene-level composition. We further introduce a voxelization-based evaluation metric capturing fine-grained geometric violations beyond 3D bounding boxes. Experiments surpass state-of-the-art on object addition and achieve superior human-perceived quality on the application of full scene synthesis, despite not being trained on it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。