评测大模型在长期规划中的持续执行能力,模拟创业公司运营一年。
$\texttt{YC-Bench}$: Benchmarking AI Agents for Long-Term Planning and Consistent Execution
- 设计模拟创业场景,评估模型跨数百步的策略一致性与适应力。
- 仅3款模型盈利超20万美元,最高达127万美元,成本仅为竞品1/11。
- 成功关键在于思维链记忆,失败主因是无法识别敌意客户(占破产47%)
随着大语言模型代理处理更复杂的任务,一个核心问题是它们能否在长周期内保持战略一致性:在不确定性下规划、从延迟反馈中学习,并在早期错误导致连锁后果时进行调整。我们提出$ exttt{YC-Bench}$,一个基准测试,要求代理在覆盖数百回合的一年模拟期内管理一家初创企业。代理需管理员工、选择任务合同,并在部分可观测环境中维持盈利,该环境存在对抗性客户和不断增长的薪酬支出,使决策失误产生累积影响。我们在3个随机种子下评估了12个模型(包括专有和开源)。仅有3个模型始终超越初始资本20万美元,其中Claude Opus 4.6平均资金达127万美元,而GLM-5以11倍更低的推理成本达到121万美元。思维链使用(scratchpad)是跨越上下文截断持久化信息的唯一机制,也是成功的最强预测因子;对抗性客户识别失败是主要失效模式,导致47%的企业破产。分析揭示前沿模型仍存在过并行化等独特失效模式,表明长周期表现仍有显著能力差距。$ exttt{YC-Bench}$ 开源、可复现且可配置。
原文摘要 · Abstract (English)
As LLM agents tackle increasingly complex tasks, a critical question is whether they can maintain strategic coherence over long horizons: planning under uncertainty, learning from delayed feedback, and adapting when early mistakes compound. We introduce $\texttt{YC-Bench}$, a benchmark that evaluates these capabilities by tasking an agent with running a simulated startup over a one-year horizon spanning hundreds of turns. The agent must manage employees, select task contracts, and maintain profitability in a partially observable environment where adversarial clients and growing payroll create compounding consequences for poor decisions. We evaluate 12 models, both proprietary and open source, across 3 seeds each. Only three models consistently surpass the starting capital of \$200K, with Claude Opus 4.6 achieving the highest average final funds at \$1.27 M, followed by GLM-5 at \$1.21 M at 11$\times$ lower inference cost. Scratchpad usage, the sole mechanism for persisting information across context truncation, is the strongest predictor of success, and adversarial client detection is the primary failure mode, accounting for $47\%$ of bankruptcies. Our analysis reveals that frontier models still fail through distinct failure modes such as over-parallelization, demonstrating the capability gaps for long-horizon performance. $\texttt{YC-Bench}$ is open-source, reproducible, and configurable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。