用强化学习提升视觉语言模型的3D空间推理能力,让虚拟场景生成更真实高效。
MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
- 通过多轮强化学习迭代优化,结合物理约束与图像评估提升布局合理性
- 在多个尺度模型上显著增强空间一致性与排布稳定性,生成结果更真实
- 适合元宇宙、AR/VR、数字孪生等需要精准3D布局的场景应用
我们提出MetaSpatial,首个基于强化学习(RL)的框架,旨在增强视觉语言模型(VLMs)的3D空间推理能力,实现无需硬编码优化的实时3D场景生成。该框架解决两大核心挑战:(i) VLMs缺乏内化的3D空间推理能力,限制其生成逼真布局;(ii) 传统监督微调(SFT)在布局生成任务中效率低下,因完美标注不可得。关键创新在于引入多轮强化学习优化机制,融合物理感知约束与渲染图像评估,确保生成的3D布局具备连贯性、物理合理性与审美一致性。方法上,MetaSpatial采用自适应、迭代式推理流程,使VLM通过分析渲染输出逐步优化空间排列。实证评估表明,该框架显著提升多种规模模型的空间一致性和格式稳定性。后训练后,物体摆放更真实、对齐且功能协调,验证了强化学习在元宇宙、AR/VR、数字孪生及游戏开发中3D空间推理的有效性。代码、数据与训练流程已公开于https://github.com/PzySeere/MetaSpatial。
原文摘要 · Abstract (English)
We present MetaSpatial, the first reinforcement learning (RL)-based framework designed to enhance 3D spatial reasoning in vision-language models (VLMs), enabling real-time 3D scene generation without the need for hard-coded optimizations. MetaSpatial addresses two core challenges: (i) the lack of internalized 3D spatial reasoning in VLMs, which limits their ability to generate realistic layouts, and (ii) the inefficiency of traditional supervised fine-tuning (SFT) for layout generation tasks, as perfect ground truth annotations are unavailable. Our key innovation is a multi-turn RL-based optimization mechanism that integrates physics-aware constraints and rendered image evaluations, ensuring generated 3D layouts are coherent, physically plausible, and aesthetically consistent. Methodologically, MetaSpatial introduces an adaptive, iterative reasoning process, where the VLM refines spatial arrangements over multiple turns by analyzing rendered outputs, improving scene coherence progressively. Empirical evaluations demonstrate that MetaSpatial significantly enhances the spatial consistency and formatting stability of various scale models. Post-training, object placements are more realistic, aligned, and functionally coherent, validating the effectiveness of RL for 3D spatial reasoning in metaverse, AR/VR, digital twins, and game development applications. Our code, data, and training pipeline are publicly available at https://github.com/PzySeere/MetaSpatial.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。