SPEAR让AI研究者用Python高效操控逼真虚拟环境,提升训练效率。
SPEAR: A Simulator for Photorealistic Embodied AI Research

- 基于Python与UE插件,可编程控制任意UE应用,功能扩展超14000个接口。
- 每秒渲染73帧1080p高清图像,速度比现有方案快一个数量级。
- 支持多智能体协同、物理模拟联动和自然语言编辑,适合复杂场景研究。
交互式模拟器已成为训练具身智能体和生成合成视觉数据的强大工具,但现有逼真模拟器存在泛化性差、可编程性弱和渲染速度慢的问题。本文提出SPEAR:一个面向逼真具身AI研究的模拟器。SPEAR是一个Python库,通过模块化插件架构连接并程序化控制任何Unreal Engine(UE)应用,向Python暴露超过14000个独特的UE函数,可编程功能量级提升一个数量级。单个SPEAR实例可在1920x1080分辨率下以73帧/秒的速度直接将逼真图像渲染至用户NumPy数组,速度较现有UE插件快一个数量级,同时提供非散射内在图像分解、材质ID、基于物理的着色参数等新真实图像模态。此外,SPEAR引入了表达力强的高层编程模型,支持定义具有任意数据依赖关系的工作图,并在单个UE帧内确定性执行。我们通过多样化应用展示了其价值:在多个野外UE项目中控制具有不同动作空间的多智能体(如人类、汽车、机器人);渲染城市规模的逼真环境;操纵UE的程序化内容生成系统;同步渲染高细节人脸的多视角图像;与MuJoCo物理模拟器进行交互共仿真;并通过AI代码助手实现自然语言编辑场景。
原文摘要 · Abstract (English)
Interactive simulators have become powerful tools for training embodied agents and generating synthetic visual data, but existing photorealistic simulators suffer from limited generality, programmability, and rendering speed. We address these limitations by introducing SPEAR: A Simulator for Photorealistic Embodied AI Research. At its core, SPEAR is a Python library that can connect to, and programmatically control, any Unreal Engine (UE) application via a modular plugin architecture. SPEAR exposes over 14K unique UE functions to Python, representing an order-of-magnitude increase in programmable functionality over existing UE-based simulators. Additionally, a single SPEAR instance can render 1920x1080 photorealistic beauty images directly into a user's NumPy array at 73 frames per second - an order of magnitude faster than existing UE plugins - while also providing ground truth image modalities that are not available in any existing UE-based simulator (e.g., a non-diffuse intrinsic image decomposition, material IDs, and physically based shading parameters). Finally, SPEAR introduces an expressive high-level programming model that enables users to specify complex graphs of UE work with arbitrary data dependencies among work items, and to execute these graphs deterministically within a single UE frame. We demonstrate the utility of SPEAR through a diverse collection of example applications: controlling multiple embodied agents with distinct action spaces (e.g., humans, cars, and robots) across several in-the-wild UE projects; rendering photorealistic city-scale environments; manipulating UE's procedural content generation systems; rendering synchronized multi-view images of detailed human faces; coordinating an interactive co-simulation with the MuJoCo physics simulator; and editing scenes with natural language via an AI coding assistant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。