自动生成线性规划题库,让大模型代理测试更公平可靠。
A$^{2}$utoLPBench: An Auto-Generated, Agent-Friendly LP Benchmark via Inverse-KKT Construction

- 通过逆KKT构造生成最优解已知的线性规划问题。
- 可无限生成新题,难度由变量和约束数控制,答案正确性由构造保证。
- 适合评估大模型代理在数学推理任务中的表现,抗训练数据泄露。
现有文本转线性规划的基准大多为人工编写的手动标注静态数据集。一旦发布,数据规模与难度固定,且所有题目可能泄露至未来大模型的训练数据中。本文提出 extbf{A$^{2}$utoLPBench},一个用于测试大模型驱动代理解决自然语言描述线性规划问题的基准。方法上,先选取一个可行点及其对偶解,再构造出以该点为最优解、目标值已知的问题。答案由构造过程保证正确,无需求解器或人工标注。评测环境包含参考求解器-评判基线与供大模型代理读取的Docker镜像,任何代理仅需一条命令即可运行并获得校准分数。由于是生成器而非固定数据集,该基准具备无限题量、可调难度(由$(n,m)$控制)、构造正确的真值答案、相比人工标注极低的每题成本、跨批次可重复得分,以及使用新种子范围时对训练数据泄露的强抵抗能力。
原文摘要 · Abstract (English)
Most LP-from-text benchmarks are static datasets of word problems written and labeled by hand. Once such a dataset is released, its size is fixed, its difficulty is fixed, and every problem can leak into the training data of future LLMs. We present \textbf{A$^{2}$utoLPBench}, a benchmark for testing LLM-driven agents on linear programming problems written in plain text. We first pick a feasible point and dual, then write down a problem for which that point is optimal and the objective value is known. The answer is known by construction, with no solver call and no human annotator. The evaluation environment bundles a reference solver-critic baseline and a Docker image whose usage instructions are written for an LLM-driven agent to read. With these in place, any agent can run the benchmark and get a calibrated score with one command. Because the benchmark is a generator rather than a fixed dataset, it has properties no fixed dataset can match: an unlimited supply of fresh problems, a difficulty knob set by $(n,m)$, ground-truth answers correct by construction, low LLM-side cost per problem relative to human authoring, repeatable scores across independent batches, and resistance to training-data leakage when fresh post-cutoff seed ranges are used.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。