用语义推理和动态路径生成沉浸式3D场景,解决物体关系与相机运动带来的生成难题。
RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roaming
- 基于视觉语言模型构建场景图,指导相机自适应移动并感知关键物体边界。
- 在合成全景数据集上微调的运动注入修复模型,可应对相机移动导致的空洞填充。
- 适合虚拟现实、游戏开发中需要真实感3D场景生成的研究与开发者。
从文本生成沉浸式3D场景是计算机视觉的核心任务,对虚拟现实与游戏开发至关重要。尽管利用2D扩散先验具有潜力,现有方法存在空间盲区,依赖预设轨迹,无法捕捉显著物体间的内在关系,因而难以理解语义布局,也无法自适应探索场景以推断被遮挡内容。此外,当前修补模型仅在2D图像空间运行,难以合理填补因相机运动产生的空洞。为此,我们提出RoamScene3D,一种连接语义引导与空间生成的新框架。该方法通过视觉语言模型(VLM)构建编码物体关系的场景图,引导相机感知显著物体边界并规划自适应漫游轨迹。为克服静态2D先验的局限,我们引入运动注入修复模型,在融合真实相机轨迹的合成全景数据集上进行微调,使其适应相机运动。大量实验表明,结合语义推理与几何约束,本方法显著优于现有最先进方法,在生成一致且逼真的3D场景方面表现优异。代码已公开于https://github.com/JS-CHU/RoamScene3D。
原文摘要 · Abstract (English)
Generating immersive 3D scenes from texts is a core task in computer vision, crucial for applications in virtual reality and game development. Despite the promise of leveraging 2D diffusion priors, existing methods suffer from spatial blindness and rely on predefined trajectories that fail to exploit the inner relationships among salient objects. Consequently, these approaches are unable to comprehend the semantic layout, preventing them from exploring the scene adaptively to infer occluded content. Moreover, current inpainting models operate in 2D image space, struggling to plausibly fill holes caused by camera motion. To address these limitations, we propose RoamScene3D, a novel framework that bridges the gap between semantic guidance and spatial generation. Our method reasons about the semantic relations among objects and produces consistent and photorealistic scenes. Specifically, we employ a vision-language model (VLM) to construct a scene graph that encodes object relations, guiding the camera to perceive salient object boundaries and plan an adaptive roaming trajectory. Furthermore, to mitigate the limitations of static 2D priors, we introduce a Motion-Injected Inpainting model that is fine-tuned on a synthetic panoramic dataset integrating authentic camera trajectories, making it adaptive to camera motion. Extensive experiments demonstrate that with semantic reasoning and geometric constraints, our method significantly outperforms state-of-the-art approaches in producing consistent and photorealistic scenes. Our code is available at https://github.com/JS-CHU/RoamScene3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。