构建动态基准测试，评估大模型组合计数能力。
CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models
- 用类型化Cofola规范生成可控制的自然语言计数题
- 11个大模型在有序/重复元素上错误率超60%
- 适合研究模型逻辑推理短板的开发者和研究人员
我们提出CombEval，一个用于评估大语言模型组合计数能力的动态基准。CombEval将每个问题表示为实体、组合对象、对象依赖和约束上的类型化Cofola规范，支持在自然语言计数题中精确生成并验证答案。相比静态数据集，CombEval可系统调节对象类型、实体规模、约束数量与推理深度。我们评估了11个大模型在直接输入与代码增强设置下的表现，发现模型在有序对象、不可区分元素、相对位置约束及嵌套依赖上仍显脆弱。错误分析揭示模型在约束理解与计数原理应用上存在根本缺陷。CombEval提供了一个诊断性测试平台，用于研究大模型在组合推理中的失败场景。代码与生成的基准套件已公开于\url{https://github.com/YuxuZhou-CN/combination-problem-generation}。
原文摘要 · Abstract (English)
We present CombEval, a dynamic benchmark for evaluating combinatorial counting in large language models. CombEval represents each problem as a typed Cofola specification over entities, combinatorial objects, object dependencies, and constraints, enabling controlled generation of natural-language counting problems with exact solver-verified answers. Unlike static collections, CombEval supports systematic variation of object type, entity scale, constraint count, and reasoning depth. We evaluate 11 LLMs under direct and code-augmented settings and find that models remain brittle on ordered objects, indistinguishable elements, relatively positional constraints, and nested object dependencies. Error analysis further identifies failures in constraint interpretation and counting principles. CombEval provides a diagnostic testbed for studying when and why LLMs fail at combinatorial reasoning. The code and generated benchmark suites are publicly available at \url{https://github.com/YuxuZhou-CN/combination-problem-generation}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。