arXiv:2605.12449cs.CV2026-05被引 1

LychSim让非图形专家也能轻松构建可控视觉仿真环境。

LychSim: A Controllable and Interactive Simulation Framework for Vision Research

论文配图:LychSim: A Controllable and Interactive Simulation Framework for Vision Research
图 1 · 摘自论文原文
  • 基于Unreal Engine 5,用Python API简化复杂引擎操作。
  • 支持生成多样高保真场景,含丰富2D/3D真实标签数据。
  • 可交互式运行大模型,适合做智能体推理与语言控制实验。

尽管自监督预训练减少了视觉系统对合成数据的依赖,但仿真仍是闭环优化和严格分布外(OOD)评估不可或缺的工具。然而,现代仿真平台通常技术门槛高,需深厚的计算机图形学与游戏开发经验。本文提出LychSim,一个基于Unreal Engine 5构建的高可控、可交互仿真框架,以弥合这一鸿沟。LychSim围绕三大设计:(1)简化版Python API,屏蔽底层引擎复杂性;(2)程序化数据流水线,可生成多样化、高保真的环境,包含不同分布外(OOD)视觉挑战,并提供丰富的2D与3D真实标签;(3)原生集成模型上下文协议(MCP),使模拟器成为推理型大模型的动态闭环试验场。我们还标注了场景级程序规则与物体级位姿对齐,实现语义一致的3D真实标签与自动场景修改。我们在多个下游任务中验证了LychSim的能力,包括作为合成数据引擎、驱动强化学习对抗性测试者,以及支持语言驱动的交互式场景布局生成。为推动社区发展,LychSim将开源,包含完整源码与各类数据标注。

原文摘要 · Abstract (English)

While self-supervised pretraining has reduced vision systems' reliance on synthetic data, simulation remains an indispensable tool for closed-loop optimization and rigorous out-of-distribution (OOD) evaluation. However, modern simulation platforms often present steep technical barriers, requiring extensive expertise in computer graphics and game development. In this work, we present LychSim, a highly controllable and interactive simulation framework built upon Unreal Engine 5 to bridge this gap. LychSim is built around three key designs: (1) a streamlined Python API that abstracts away underlying engine complexities; (2) a procedural data pipeline capable of generating diverse, high-fidelity environments with varying out-of-distribution (OOD) visual challenges, paired with rich 2D and 3D ground truths; and (3) a native integration of the Model Context Protocol (MCP) that transforms the simulator into a dynamic, closed-loop playground for reasoning agentic LLMs. We further annotate scene-level procedural rules and object-level pose alignments to enable semantically aligned 3D ground truths and automated scene modification. We demonstrate LychSim's capability across multiple downstream applications, including serving as a synthetic data engine, powering reinforcement learning-based adversarial examiners, and facilitating interactive, language-driven scene layout generation. To benefit the broader vision community, LychSim will be made publicly available, including full source code and various data annotations.

视觉仿真可控生成大模型交互游戏引擎

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。