测试AI在规则突变下的适应能力,揭示其真实智能水平
MirrorCraft: Paired Evaluation under Hidden Rule Changes in Minecraft

- 构建配对世界,隐藏修改游戏规则以模拟真实变化
- 规则变更影响显著,部分配置表现下降超40%
- 适合评估AI在动态环境中的泛化与推理能力
随着大语言模型(LLMs)的发展,研究基于LLM的代理在Minecraft中的表现成为热点。然而,现有基准大多在固定游戏机制下评估,无法检验代理在熟悉规则改变时是否仍能持续进步。本文提出MirrorCraft,一个配对基准,用于评估代理在隐藏规则变化下的表现。每个镜像世界(Mirror)是对应原版世界(Vanilla)的副本,仅通过数据包修改部分服务器端规则。地形、生成点、资源分布、目标、界面和动作预算在每对世界间保持一致。该基准包含五个受控生物群系、六套规则、三个进展目标、两个模型家族和六种代理配置,均基于统一的Mineflayer接口。使用确定性里程碑和成功率衡量任务进展,并引入规则干预效应(RIE)量化匹配世界间的性能差异。实验表明,规则变化对不同规则套件影响各异;在未提供规则描述的情况下,ReAct获得最高综合镜像得分;提供确切规则可小幅提升平均进度与完成率。MirrorCraft将Minecraft评估拓展至非固定机制场景,为研究代理在规则差异下的行为提供了受控环境。
原文摘要 · Abstract (English)
With the prosperity of the large language models (LLMs), it has become an interesting topic: how do LLM-based agents work in Minecraft? Unfortunately, most existing benchmarks evaluate them under fixed game mechanics. High performance in these settings does not show whether an agent can continue making progress when familiar recipes, drops, and other rules change. In this paper, we introduce MirrorCraft, a paired benchmark for evaluating agents under hidden rule changes in Minecraft. Each Mirror world is a copy of its paired Vanilla world, with selected server-side rules modified by the corresponding datapack. Terrain, spawn, resource placement, objective, interface, and action budget remain matched within every Vanilla-Mirror pair. MirrorCraft includes five controlled biomes, six rule suites, three progression objectives, two model families, and six agent configurations under a shared Mineflayer interface. We evaluate task progress with deterministic advancement milestones and success rate and use the Rule Intervention Effect (RIE) to measure the performance change between matched Vanilla and Mirror worlds. The experiments show that hidden rule changes have strongly different effects across suites. Among the configurations evaluated without rule descriptions, ReAct achieves the highest pooled Mirror score. Providing the exact rules yields modest gains in average progress and completion across all three objectives. MirrorCraft extends Minecraft evaluation beyond fixed mechanics and provides a controlled setting for studying how agents use gameplay outcomes when the rules of the current world differ from familiar ones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。