测试大模型直接求解带约束优化问题的能力,发现可行性是主要瓶颈。
ConstraintBench: Benchmarking LLM Constraint Reasoning on Direct Optimization
- 构建10个运筹学领域的真实场景,要求模型直接输出满足所有约束的解
- 最佳模型仅65%解可行,可行解平均达最优值的89%~96%
- 适合关注大模型决策能力与可靠性评估的研究者
大语言模型在需要约束优化的运营决策中应用日益广泛。现有基准仅评估模型能否将优化问题转化为求解器代码,但未解决关键问题:模型能否在无求解器支持下直接给出正确解?我们提出ConstraintBench,一个涵盖10个运筹学领域的基准,所有真实解均由Gurobi求解器验证。每项任务以自然语言描述场景,包含实体、约束和目标函数;模型需返回结构化解,由确定性验证器检查是否满足全部约束及求解器最优值。我们在200个任务上评估六种前沿模型,发现可行性是主要瓶颈——最佳模型仅达65.0%可行性,而可行解平均为Gurobi最优值的89%至96%。无一模型在0.1%误差内同时实现可行性与最优性,最高仅30.5%。各领域差异显著,设施选址平均可行性达85.0%,而乘务员排班仅0.8%。系统性失败模式包括时长约束误解、实体幻觉,以及在设施选址与车辆路径问题中出现的可行性与最优性解耦现象。
原文摘要 · Abstract (English)
Large language models are increasingly applied to operational decision-making where the underlying structure is constrained optimization. Existing benchmarks evaluate whether LLMs can formulate optimization problems as solver code, but leave open a complementary question. Can LLMs directly produce correct solutions to fully specified constrained optimization problems without access to a solver? We introduce ConstraintBench, a benchmark for evaluating LLMs on direct constrained optimization across 10 operations research domains, with all ground-truth solutions verified by the Gurobi solver. Each task presents a natural-language scenario with entities, constraints, and an optimization objective; the model must return a structured solution that a deterministic verifier checks against every constraint and the solver-proven optimum. We evaluate six frontier models on 200 tasks and find that feasibility, not optimality, is the primary bottleneck. The best model achieves only 65.0% feasibility, yet feasible solutions average 89 to 96% of the Gurobi-optimal objective. No model exceeds 30.5% on joint feasibility and optimality within 0.1% of the solver reference. Per-domain analysis shows large variation in difficulty, with average feasibility spanning from 85.0% in the facility location domain to 0.8% in the crew assignment domain. Further, systematic failure modes include duration constraint misunderstanding, entity hallucination, and a feasibility-optimality decoupling in facility location and vehicle routing where models achieve high feasibility but 0% optimality. ConstraintBench and all evaluation infrastructure will be publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。