首个评估大模型环境模拟能力的基准，揭示其在状态更新时易出错。
EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation
- 定义并量化环境模拟能力，构建覆盖167个环境的基准测试集
- 发现主流大模型在多状态同步更新时准确率骤降，暴露关键能力短板
- 提出约束驱动管道，减少幻觉，提升生成效率6.8%，成本降低90%以上
可扩展的AI智能体训练依赖于能真实反映智能体行为后果的交互环境。人工构建环境成本高、难以扩展且多样性有限。一种有前景的方向是用大模型模拟环境替代人工设计。但该范式依赖一个未经验证的核心假设：大模型能准确模拟环境反馈。实践中，大模型模拟环境存在幻觉、逻辑矛盾和隐性状态漂移等问题，破坏奖励信号并加剧构建成本。为此，我们提出EnvSimBench，包含四项贡献：1）首次正式定义并操作化环境模拟能力（EnvSim Ability）为可量化的研究目标；2）构建涵盖400个样本、167个多样化环境的严格基准，配备可验证标签与三轴细粒度难度分层；3）系统评估显示，所有先进语言模型均存在普遍的状态变更悬崖现象：当环境状态不变时表现接近完美，但需同时更新多个状态时则严重失败；这一发现揭示了环境模拟能力是关键却长期被忽视的能力缺口；4）设计约束驱动的模拟流程，显著降低幻觉，环境合成产出率提升6.8%，成本下降超90%。总体而言，EnvSimBench既是诊断框架，也是实现可靠大模型环境模拟的实用优化路径，为可扩展智能体训练奠定基础。代码与数据见https://github.com/cookieApril/EnvSimBench。
原文摘要 · Abstract (English)
Scalable AI agents training relies on interactive environments that faithfully simulate the consequences of agent actions. Manually crafted environments are expensive to build, brittle to extend, and fundamentally limited in diversity. A promising direction is to replace manually crafted environments with LLM-simulated counterparts. However, this paradigm hinges on an unexamined core assumption: LLMs can accurately simulate environmental feedback. In practice, LLM-simulated environments suffer from hallucinations, logical inconsistencies, and silent state drift failures that corrupt agent reward signals and compound the construction costs that the paradigm was designed to eliminate. To address this gap, we propose EnvSimBench with four contributions: 1) We provide the first formal definition and operationalization of Environment Simulation Ability (EnvSim Ability) as a quantifiable research objective. 2) We construct EnvSimBench, a rigorous benchmark covering 400 samples across 167 diverse environments, equipped with verifiable labels and fine-grained difficulty stratification along three axes. 3) Systematic evaluations reveal that all state-of-the-art language models suffer from a universal state change cliff: they achieve near-perfect accuracy on tasks when the environment state remains invariant, yet fail catastrophically when multiple states need simultaneous updates. This finding exposes EnvSim Ability as a critical yet largely unaddressed capability gap. 4) We design a constraint-driven simulation pipeline that substantially reduces hallucination, boosts environment synthesis yield by 6.8%, and cuts costs by over 90%. Overall, EnvSimBench serves as both a diagnostic framework and a practical optimization path for reliable LLM-based environment simulation, establishing a foundation for scalable agent training. Code and data are available at https://github.com/cookieApril/EnvSimBench
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。