用语言模型规划+视觉引导,生成更真实可交互的3D场景。
Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation
- 先用大模型生成粗略布局,再用视觉模块优化空间关系。
- 通过迭代优化实现物体不穿透、姿态准确,提升物理合理性。
- 无需训练,适合虚拟内容创作与智能体仿真研究。
从文本生成可交互的3D场景对游戏、虚拟现实和具身AI至关重要。现有方法面临挑战:基于学习的方法依赖小规模室内数据集,限制了场景多样性与布局复杂性;而大型语言模型(LLMs)虽能利用多样化文本知识,却难以保证空间合理性,常出现不符合常识的物体摆放。本文提出Scenethesis,一个无需训练的智能体框架,将LLM驱动的场景规划与视觉引导的布局优化相结合。给定文本提示后,首先由LLM生成粗略布局;随后视觉模块通过生成图像引导并提取场景结构,捕捉物体间关系;接着优化模块迭代调整物体位姿与物理合理性,防止穿透与不稳定;最后判别模块验证空间一致性。全面实验表明,Scenethesis能生成多样、真实且物理合理的3D交互场景,适用于虚拟内容生成、仿真环境构建及具身AI研究。
原文摘要 · Abstract (English)
Synthesizing interactive 3D scenes from text is essential for gaming, virtual reality, and embodied AI. However, existing methods face several challenges. Learning-based approaches depend on small-scale indoor datasets, limiting the scene diversity and layout complexity. While large language models (LLMs) can leverage diverse text-domain knowledge, they struggle with spatial realism, often producing unnatural object placements that fail to respect common sense. Our key insight is that vision perception can bridge this gap by providing realistic spatial guidance that LLMs lack. To this end, we introduce Scenethesis, a training-free agentic framework that integrates LLM-based scene planning with vision-guided layout refinement. Given a text prompt, Scenethesis first employs an LLM to draft a coarse layout. A vision module then refines it by generating an image guidance and extracting scene structure to capture inter-object relations. Next, an optimization module iteratively enforces accurate pose alignment and physical plausibility, preventing artifacts like object penetration and instability. Finally, a judge module verifies spatial coherence. Comprehensive experiments show that Scenethesis generates diverse, realistic, and physically plausible 3D interactive scenes, making it valuable for virtual content creation, simulation environments, and embodied AI research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。