用20个世界标记实现高效3D场景建模,让机器人更准地完成语言指令操作。
GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation

- 在VLA模型中插入状态与预测标记,直接生成共享高斯基元的当前与未来场景
- 训练时98.6%成功率(LIBERO),推理仅需20个令牌,无在线解码开销
- 适合需要高效闭环控制的现实机器人任务,尤其在视角和布局变化下表现稳健
视觉-语言-动作(VLA)策略推动了语言驱动的机器人操作发展,但动作模仿目标对度量3D结构和短时物理演化提供弱监督。现有几何增强策略侧重当前场景定位,而预测类策略常在RGB或隐空间建模未来动态,部署成本高。GaussianDream表明,训练时对当前高斯重建与未来高斯预测可提供有效3D监督,但其密集的VGGT/TGE前缀同时携带状态、动力学与动作条件信息。本文提出一种紧凑的、原生集成于策略中的扩展方法,在VLA主干中直接插入“世界状态标记”与“世界预测标记”。一个仅训练使用的“世界表示头”将这些标记解码为共享高斯基元上的当前世界与耦合未来预测,通过静态-动态分解保留持久结构,并聚焦残差运动于交互相关区域。推理阶段移除表示头、渲染器、辅助目标及VGGT/TGE路径,仅保留20个世界标记,无需在线高斯解码或滚动预测。该方法在LIBERO上达到98.6%成功率,在LIBERO-Plus上达87.8%,在相机与布局变换下仍保持显著优势。真实机器人实验显示,平均成功率从29.2%提升至52.5%,相比复现的π₀.₅策略,且维持高效闭环控制。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) policies have advanced language-conditioned robotic manipulation, yet action-imitation objectives provide only weak supervision for metric 3D structure and short-horizon physical evolution. Geometry-enhanced policies mainly improve current-scene grounding, whereas predictive policies often model future dynamics in RGB or latent spaces and may incur substantial deployment cost. GaussianDream demonstrates that training-time current Gaussian reconstruction and future Gaussian prediction provide effective 3D supervision, but its dense VGGT/TGE-based prefix jointly carries state, dynamics, and action-conditioning information. We present \textbf{\methodname}, a compact, policy-native extension that inserts \textbf{World State Tokens} and \textbf{World Prediction Tokens} directly into the VLA backbone. A training-only \textbf{World Representation Head} decodes these tokens into a Current World and coupled Future Prediction over shared Gaussian primitives, while static--dynamic factorization preserves persistent structure and focuses residual motion on interaction-relevant regions. At inference, the head, renderer, auxiliary objectives, and VGGT/TGE pathway are removed, leaving only 20 world tokens without online Gaussian decoding or rollout. \method achieves \textbf{98.6\%} on LIBERO and \textbf{87.8\%} on LIBERO-Plus, with clear gains under Camera and Layout shifts. Real-robot experiments further improve average success from 29.2\% to 52.5\% over reproduced $π_{0.5}$ while maintaining efficient closed-loop control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。