测试大模型代理解决全流程运筹学问题的能力,发现当前表现仍不靠谱。
ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?

- 构建可执行的运筹学任务基准,覆盖107个真实场景
- 最优代理仅通过35.51%任务,难题通过率仅20.59%
- 适合关注智能决策系统可靠性的研究者
大型语言模型正被用于在可执行环境中完成多步任务,但其在真实运筹学(OR)工作中的能力尚不明确。现有评估常将建模与求解分离,依赖预定义或纯文本实例,且极少测试从操作文档到验证决策的完整流程。本文提出ORAgentBench,一个面向自主代理的、基于执行的端到端运筹学任务评估基准。该基准包含107个经人工审核的任务,涵盖多样化的运营场景,每个任务均封装在独立环境中,包含自然语言说明、多文件数据、配置文件和提交格式要求。代理需编写并运行求解代码,其提交结果由隐藏验证器评估是否符合格式、满足硬约束及目标函数质量。对14种前沿代理-模型组合的实验显示,当前代理远未达到可靠的运筹实践水平:最优代理仅通过35.51%的任务,难题通过率仅为20.59%,且许多可行解仍低于质量阈值。失败分析表明,主要错误源于策略性弱点,包括遗漏运营规则、公式脆弱、可行解构造薄弱及解改进不足。针对运筹学的程序化技能虽能提升难题通过率,但无法稳定提高解质量或通过率。结果表明,运筹学代理的发展需超越生成合理优化代码,转向可信赖、高质量的运营决策能力。
原文摘要 · Abstract (English)
Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear. Existing OR evaluations often decouple modeling from solving, rely on pre-formalized or text-only instances, and rarely test the full workflow from operational artifacts to validated decisions. In this work, we introduce ORAgentBench, an execution-grounded benchmark for evaluating autonomous agents on challenging end-to-end operations research tasks. It contains 107 human-reviewed tasks across diverse operational scenarios, each packaged in an isolated environment with a natural-language brief, multi-file data, configuration artifacts, and a required submission schema. Agents must write and run solution code, and their submissions are evaluated by hidden validators for schema validity, hard-constraint feasibility, and normalized objective quality. Experiments with fourteen frontier agent-model configurations show that current agents remain far from reliable OR practice. The best agent passes only 35.51% of all tasks and 20.59% of hard tasks, and many feasible submissions still fall below the required quality threshold. Failure analysis further shows that errors are dominated by strategic weaknesses, including missed operational rules, brittle formulations, weak feasible-solution construction, and insufficient solution improvement. OR-specific procedural skills increase hard-task feasibility, but do not reliably improve solution quality or pass rate. These results suggest that progress in OR agents requires moving beyond plausible optimization code toward dependable, high-quality operational decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。