用视觉语言智能体自动把真实场景转成可运行的物理仿真,省去人工调参。
Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents

- 通过视觉语言代理自动完成几何重建、物理参数推断和仿真环境组装。
- 在刚体、柔体和人形运动场景中均实现高成功率转换,覆盖多个传统独立流程领域。
- 支持低成本开源模型,适合机器人策略训练与评估等下游任务使用。
真实世界到仿真环境的转换在机器人与物体交互中仍需大量人工干预,不仅需要视觉重建,还需恢复场景几何、物体状态、推断物理参数,并整合执行器、物体、相机、位姿和轨迹以构建可运行的物理仿真。当前流程依赖视觉基础模型的手动调参、网格清理、坐标对齐及感知工具与仿真器间的脆弱衔接。我们提出 extit{Agentic Real2Sim},一种基于视觉语言代理的通用物理世界建模框架,将真实世界中的物体-机器人交互视频转化为可模拟的事件孪生体,保留观测数据、几何结构、机器人交互和物体状态。我们在刚体操作、柔体交互及人形运动场景中进行评估,涵盖通常由不同独立管道处理的领域,标志着向可扩展转换迈出第一步。该框架的代理决策可由开权重视觉语言模型驱动,成本仅为前沿模型的小部分,同时达到相当的转换成功率。目标是利用生成的与真实对齐的孪生体,用于机器人策略学习与评估等下游任务。
原文摘要 · Abstract (English)
Real-to-sim conversion for robotic interaction with objects remains labor-intensive because it requires more than visual reconstruction: a streamlined real2sim process must recover scene geometries and object states, infer physical parameters, and assemble actors, objects, cameras, poses, and trajectories into a runnable physical simulation. Today this process still depends on manual tuning of visual foundation models, mesh cleanup, coordinate-frame alignment, and brittle workflow glue across visual perception tools and simulators. We introduce \textit{Agentic Real2Sim}, a framework for generalized physical world modeling with vision-language agents, converting a real-world recording of object-robot interaction into a simulatable episodic twin which preserves observations, geometries, robot interactions, and object states. We evaluate Agentic Real2Sim on rigid-object manipulation, deformable-object interaction, and humanoid motion scenes, spanning domains that are usually handled by separate Real2Sim pipelines, marking a first step toward scalable conversion. The framework's agentic decisions can be driven by an open-weight VLM backend at a small fraction of the cost of frontier models, while attaining comparable conversion success rate. We aim to use the resulting real-world-aligned twins for downstream robotics tasks, specifically policy learning and evaluation. The project site is available at https://agentic-real2sim.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。