用新基准评估大模型对组合优化问题的建模能力,最高准确率达91%。
DCP-Bench-Open: Evaluating LLMs for Constraint Modelling of Discrete Combinatorial Problems
- 构建跨领域组合问题基准,支持多框架建模评测
- 高阶Python框架下模型表现最佳,准确率超91%
- 提示工程与推理计算提升效果,适合工业优化研究者
离散组合问题(DCPs)广泛存在于工业决策与优化中。尽管约束求解技术进步显著,但其核心环节——约束建模仍依赖专家经验,成为广泛应用的瓶颈。近期研究尝试用大语言模型(LLMs)将问题描述转化为可执行约束模型。然而现有评估数据集多局限于小规模、同质化或特定领域问题,难以反映真实场景多样性。本文提出DCP-Bench-Open,一个源自约束编程(CP)与运筹学(OR)社区的多样化离散组合问题基准,专为评估LLM驱动的建模能力而设计。在该数据集上,我们对比了三种不同抽象层级和语法结构的建模系统,发现基于高阶Python框架的建模表现更优。同时系统评估了提示工程与推理时计算方法的效果,进一步提升准确性,最高达91%。该基准已公开。
原文摘要 · Abstract (English)
Discrete Combinatorial Problems (DCPs) are prevalent in industrial decision-making and optimisation. However, while constraint solving technologies for DCPs have advanced significantly, the core process of formalising them, namely constraint modelling, requires significant expertise and remains a bottleneck for wider adoption. Aiming to alleviate this bottleneck, recent studies have explored using Large Language Models (LLMs) to transform combinatorial problem descriptions into executable constraint models. However, the existing evaluation datasets for discrete constraint modelling are often limited to small, homogeneous, or domain-specific problems, which do not capture the diversity of real-world scenarios. This work addresses this gap by introducing DCP-Bench-Open, a novel benchmark that includes a diverse set of well-known discrete combinatorial problems sourced from the Constraint Programming (CP) and Operations Research (OR) communities, structured explicitly for evaluating LLM-driven constraint modelling. With this dataset, and given the variety of modelling frameworks, we compare and evaluate the modelling capabilities of LLMs for three distinct constraint modelling systems, which vary in abstraction level and underlying syntax. Notably, the results show higher performance when modelling with a high-level Python-based framework. Additionally, we systematically evaluate the use of prompt-based and inference-time compute methods across different LLMs, which further increase accuracy, reaching up to 91% on this highly challenging benchmark. DCP-Bench-Open is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。