评测大模型设计高效优化算法的能力,揭示其在真实大规模问题上的短板。
FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization

- 构建180个来自顶会论文的真实优化任务,评估模型设计高效算法能力。
- 顶尖模型仅31%任务优于Gurobi,进化型代理在难题上达成50%性能提升。
- 适合关注大模型优化生成、算法设计能力评估的研究者使用。
大型语言模型(LLMs)被广泛用于优化建模与求解器代码生成,但实际运筹学问题更需设计能利用问题结构、超越直接求解基线的可扩展算法。现有基准多限于小规模或简化案例,远低于现实复杂度。我们提出FrontierOR,首个系统评估大模型在真实大规模优化中设计高效算法能力的基准。该基准包含180项任务,源自顶级运筹学期刊发表的方法多样论文,每项配有标准化实例和专家验证的隐藏评测集。我们评估了七种涵盖前沿、成本效益及开源模型的LLMs,在单次提示与测试时演化两种设置下表现。结果显示,即使最强大的模型也难以从可执行公式迈向高效算法:最强单次提示模型在解质量与计算效率上仅31%任务优于Gurobi;即便强编码代理采用测试时演化,也仅在部分难题上达到50%性能。FrontierOR为大模型优化算法设计提供实用评估平台,推动未来模型与智能体系统性检验其能否从正确建模跃升至可行、高质量、高效的算法。代码与数据已公开于https://github.com/Minw913/FrontierOR。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimization problems often require a harder capability: designing scalable algorithms that exploit problem structure and outperform direct formulation-and-solve baselines. Existing benchmarks are limited to small or simplified examples far below real-world scale and complexity. We introduce FrontierOR, among the first benchmarks to systematically evaluate LLM-based efficient algorithm design for realistic large-scale optimization problems. FrontierOR includes 180 tasks derived from methodologically diverse papers published in top-tier operations research venues, each with standardized instances and a hidden, expert-verified evaluation suite. We evaluate seven LLMs spanning frontier, cost-effective, and open-source models both in one-shot and test-time evolution settings. The results reveal that frontier models still struggle to move from executable formulations to efficient optimization algorithms: the strongest one-shot model outperforms Gurobi in only 31% of cases in both solution quality and computational efficiency, and even strong coding agents with test-time evolution achieve only 50% on selected hard tasks. FrontierOR establishes a practical evaluation platform for LLM-based optimization algorithm design, which enables future LLMs and agents to be systematically tested on whether they can move beyond correct formulation toward a feasible, high-quality, and efficient algorithm. Code and data are publicly released at https://github.com/Minw913/FrontierOR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。