构建可自适应变难的组合推理测试集，评估大模型真实推理能力。
MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMs
- 用可扩展的约束问题生成器动态制造难题，防止模型快速过拟合。
- 在329个实例上，顶级模型准确率从66.9%降至18.5%，证明基准抗进步能力。
- 工具调用显著提效（最高+52百分点），且对调用次数敏感，适合研究工具使用。
我们提出MathConstraint，一个硬性、自适应的基准，用于评估大模型的组合推理能力。通过将约束满足问题与严格求解器验证结合，设计自适应生成器，确保问题难度随模型进步持续挑战。不同于固定数据集或依赖大模型自判的基准，MathConstraint采用参数化问题类型，支持生成任意复杂且自动可验证的实例。我们发布MathConstraint-Easy（266个实例），前沿模型准确率在72.6%（gemini-3.1-flash-lite）至87.6%（gpt-5.5）之间；MathConstraint（329个实例）上，同模型准确率降至18.5%（claude-4.6-sonnet）至66.9%（gpt-5.5），体现其对模型进步的鲁棒性。评估12个前沿及开源模型，带沙箱内Python环境（含通用SAT/SMT求解器）可使平均准确率提升28个百分点（最高+52pp）。将工具调用预算从8次减至4次，准确率最多下降37个百分点——这一敏感性是多数单预算基准所忽略的。我们开放生成器、数据集和评测框架，为组合推理与工具使用行为研究提供可靠环境。
原文摘要 · Abstract (English)
We introduce MathConstraint, a hard, adaptive benchmark for evaluating the combinatorial reasoning capabilities of LLMs. We combine constraint satisfaction problems with rigorous solver-based verification and design an adaptive generator to create instances that remain challenging as the LLMs improve in their reasoning capabilities. Unlike existing benchmarks that quickly saturate on fixed datasets or use LLM-as-a-judge for checking solutions,MathConstraint uses parameterized problem types that enable scalable generation of arbitrarily difficult and automatically verifiable instances. We release MathConstraint-Easy ($266$ instances), on which frontier models achieve between $72.6\%$ (gemini-3.1-flash-lite) and $87.6\%$ (gpt-5.5) accuracy, and MathConstraint ($329$ instances) on which the same models drop to between $18.5\%$ (claude-4.6-sonnet) and $66.9\%$ (gpt-5.5) accuracy, demonstrating the resilience of our benchmark generator against rapid progress in LLM reasoning capabilities. We evaluate 12 frontier and open-weight models with and without access to a sandboxed Python environment that includes generic SAT/SMT solvers. Tool access roughly doubles frontier accuracy on MathConstraint (mean $+28$pp; up to $+52$pp for claude-4.6-sonnet). Further, halving the tool-call budget from $8$ to $4$ rounds erases up to $37$ points -- a sensitivity that most single-budget benchmarks miss. We release the generator, dataset, and evaluation harness as a robust environment for studying combinatorial reasoning and tool-use behavior under adversarially-tunable difficulty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。