评测大模型在持续经济环境中的长期规划与执行能力
EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies
- 构建三个开源经济环境，支持超长周期（1000+步）连续决策
- 多模型测试显示无一在所有场景中全优，策略与执行均存短板
- 适合研究长期智能体策略、经济模拟及可控性权衡的学者
长期规划是自主大模型代理的核心能力，但现有评估框架多为短期、特定领域或缺乏持久经济动态支撑。本文提出EcoGym，一个通用的连续计划与执行决策基准，涵盖三个多样化环境：自动售货机（Vending）、自由职业（Freelance）和运营（Operation），均采用统一决策流程与标准接口，并支持预算约束下的超长周期运行（1000+步，相当于365天循环）。评估基于商业相关指标（如净财富、收入、日活用户数），关注长期战略连贯性与部分可观测性、随机性下的鲁棒性。对十一个主流大模型的实验表明，无一模型在所有场景中占优，普遍存在高层策略或执行效率不足的问题。EcoGym作为开放可扩展的测试平台，支持透明的长期智能体评估及经济场景中可控性与效用的权衡研究。
原文摘要 · Abstract (English)
Long-horizon planning is widely recognized as a core capability of autonomous LLM-based agents; however, current evaluation frameworks suffer from being largely episodic, domain-specific, or insufficiently grounded in persistent economic dynamics. We introduce EcoGym, a generalizable benchmark for continuous plan-and-execute decision making in interactive economies. EcoGym comprises three diverse environments: Vending (adapted from the closed-source Vending-Bench, with full open-source release), Freelance (new), and Operation (new), implemented in a unified decision-making process with standardized interfaces, and budgeted actions over an effectively unbounded horizon (1000+ steps if 365 day-loops for evaluation). The evaluation of EcoGym is based on business-relevant outcomes (e.g., net worth, income, and DAU), targeting long-term strategic coherence and robustness under partial observability and stochasticity. Experiments across eleven leading LLMs expose a systematic tension: no single model dominates across all three scenarios. Critically, we find that models exhibit significant suboptimality in either high-level strategies or efficient actions executions. EcoGym is released as an open, extensible testbed for transparent long-horizon agent evaluation and for studying controllability utility trade-offs in economic settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。