用混合智能体进化生成高效多样的仿真场景,速度快24倍且更真实。
SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

- 先用图像先验快速生成初始场景,再用视觉语言模型智能体迭代优化。
- 在SceneEval-100上达最强基线的布局质量,速度提升24倍,物理错误显著减少。
- 适合需要快速生成多样仿真环境的交互娱乐与具身AI研究者。
多样且可直接用于仿真的室内场景对互动娱乐和具身AI至关重要,但其规模化生成仍具挑战。现有基于视觉语言模型(VLM)的智能体文本转3D场景方法虽能生成高保真场景,但需昂贵的迭代物体摆放与优化;而基于参数化图像到3D的模型虽效率高,却常导致不精确且物理上无效的场景。更重要的是,二者均难以为单一输入生成多样化结果,无法反映真实场景的动态变化。本文提出 extbf{SceneMosaic},融合两者优势:从图像先验中获取初始候选,再通过VLM智能体演化优化,兼顾效率与物理合理性。该方法利用自然场景的局部性,将场景分解为独立局部单元,分别演化后再通过笛卡尔积组合成全局场景。在SceneEval-100上,SceneMosaic在语义布局质量上媲美最强的智能体基线,实现24倍加速,显著降低物理违规,并获得最高人类评分。代码已开源于https://github.com/rxjfighting/SceneMosaic。
原文摘要 · Abstract (English)
Diverse and simulation-ready indoor scenes are essential for interactive entertainment and embodied AI, yet their scalable generation remains challenging. Recent agentic text-to-3D scene pipelines that rely on vision-language models (VLMs) can generate scenes of high fidelity but require costly iterative object placement and refinement. Another mainstream paradigm, parametric image-to-3D scene models, produces scenes efficiently from strong priors learned from 2D images but often leads to imprecise and physically invalid scenes. More importantly, both paradigms struggle to output diverse scenes for a single input, making it hard for them to reflect the dynamically changing nature of real scenes. In this paper we propose \textbf{SceneMosaic}, a framework that combines the merits of both paradigms. It obtains the initial candidate from the learned image-based prior, and subsequently evolves the result through VLM agents, ensuring both efficiency and physical validity. Within the evolution process, SceneMosaic exploits the locality of natural scenes and decomposes a scene into independent local units, allowing separate evolution within each unit before composing the global scene via Cartesian product. On SceneEval-100, SceneMosaic matches the strongest agentic baseline in semantic layout quality with a 24x speedup, substantially reduces physical violations, and receives the highest human ratings. Our code is publicly available at https://github.com/rxjfighting/SceneMosaic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。