统一机器人仿真框架,支持任务评估与自动执行
MagicSim: A Unified Infrastructure for Executable Embodied Interaction

- 基于统一确定性运行时构建可执行世界,解耦内容与行为配置
- 单个任务定义支持评测、自动轨迹生成与交互接口,支持多模态轨迹保存
- 适用于强化学习、具身智能与视觉语言模型的交互实验
机器人学习与具身智能如今需要仿真作为连接控制、技能与规划的共享执行基础,而不仅是渲染器或固定任务环境。现有流程常通过‘魔法’操作割裂各层,训练环境脱节,或仅支持单向渲染,无法复现、评估和标注同一剧集。我们提出 MagicSim,一种围绕单一确定性批量运行时和共享马尔可夫决策过程(MDP)构建的具身交互基础设施。通过 YAML 首次配置,解耦内容、布局、行为与智能体暴露,构建涵盖任务族、交互模式、物理特性、布局、传感器、化身与机器人形态的多样化可执行世界。统一执行接口通过控制器、原子技能、规划原语和异步规划实现高阶命令,转化为机器人动作而非模拟器状态修改。单一任务定义支持三项能力:基准测试与强化学习评估、自动生成轨迹的自动采集接口,以及面向智能体与视觉语言模型的交互。自动执行中,命令经由命令→技能→规划→机器人→记录流水线处理,各环境的命令、技能、规划、重试、标注与剧集状态独立演进,高于共享物理时钟。成功回放以结构化多模态轨迹形式保存,对齐语言监督、动作表示、视觉/几何表征与任务状态。MagicSim 在一个规划闭环运行时中,统一了世界构建、具身执行、任务评估、自动回放生成与交互接口。
原文摘要 · Abstract (English)
Robot learning and embodied agents now require simulation to serve as a shared execution substrate linking control, skills, and planning, not only as a renderer, controller testbed, or fixed task environment. Existing pipelines split these layers with "magic" actions, disconnected training environments, or forward-only renders that cannot reproduce, evaluate, and annotate the same episode. We present MagicSim, an embodied interaction infrastructure built around one deterministic batched runtime and a shared Markov decision process (MDP). From YAML-first specifications that decouple contents, placement, behavior, and agent exposure, MagicSim constructs diverse executable worlds spanning task families, interaction regimes, physics, layouts, sensors, avatars, and robot embodiments in one reset-and-step loop. A common execution interface grounds high-level commands through controllers, atomicskills, planner primitives, and asynchronous planning, realizing them as robot actions rather than simulator-side state edits. One task definition supports three capabilities: benchmark and RL evaluation, an autocollect interface that automatically turns commands into grounded trajectories, and agent/VLM-facing interaction. For automatic execution, commands flow through a Command->Skill->Planner->Robot->Record pipeline, while per-environment command, skill, planning, retry, annotation, and episode states advance independently above the shared physics tick. Successful rollouts are saved as structured multimodal trajectories aligning language supervision, action representations, visual/geometric representations, and task-level status with the executed episode. MagicSim thus unifies diverse world construction, embodied execution, task evaluation, automatic rollout generation, and interactive agent interfaces in one planner-in-the-loop runtime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。