arXiv:2608.14999cs.CLcs.AI2026-08

构建70个未被充分研究的图构造难题,测试大模型推理能力。

RamseyGadgets: A Graph Construction Dataset for LLMs

论文配图:RamseyGadgets: A Graph Construction Dataset for LLMs
图 1 · 摘自论文原文
  • 设计70个需特定性质的图构造问题,验证大模型推理而非记忆。
  • 大模型在难题上平均准确率仅37.7%,最高为Gemma-4-31B。
  • 问题规模小(≤10顶点),可自动验证,适合评估与扩展。

在图论与计算机科学中,构造特殊图是一项重要任务。许多经典图构造源于对相关图的全面探索和人类智慧。随着生成式AI在数学中的兴起,自然要检验大语言模型是否能利用其推理能力构造满足特定性质的图。然而,许多自然的图构造问题(如寻找极值Ramsey好图,即避免特定单色子图)已在文献中被广泛研究,难以判断构造结果是来自模型推理还是训练数据的回忆。本文提出 extbf{RamseyGadgets},一个包含70个未被充分研究的图构造问题的新数据集,要求找到具有特殊性质的Ramsey好图(例如:包含固定颜色的边)。这些问题的解规模较小(最多10个顶点),可用SAT求解器验证,适合自动评估。该数据集易于扩展,只需改变所避单色子图即可生成新问题。我们评估了五种开源大模型在该数据集上的表现,结果显示模型在难题上的平均准确率为37.70%,其中Gemma-4-31B表现最佳。此外,我们还展示了该数据集如何揭示哪些提示能提升模型表现。

原文摘要 · Abstract (English)

Constructing special graphs is an important task within graph theory and computer science. Many popular graph constructions are the result of a comprehensive exploration of relevant graphs and human ingenuity. Given the rise of generative AI usage in mathematics, it is natural to test whether LLMs are able to construct graphs with specified properties using their reasoning capabilities. Unfortunately, many natural graph construction problems, such as finding extremal Ramsey-good graphs (i.e., avoiding specific monochromatic subgraphs), have been explored extensively in the literature, making it difficult to ascertain whether a construction is the product of an LLM's reasoning capabilities or its recollection from training data. In this work, we introduce \textbf{RamseyGadgets}, a novel dataset of 70 underexplored graph construction problems that require finding Ramsey-good graphs with special properties (e.g., containing an edge with a fixed color). These problems have reasonably sized solutions (at most 10 vertices) that can be verified by SAT solvers, making them suitable for automatic evaluation. Our dataset is easily expandable, as one can simply change the monochromatic subgraphs being avoided to obtain a new set of problems. We evaluate the performance of five open-source LLMs on our dataset and report the results. Our findings show that LLMs achieve only 37.70% accuracy on the hard-tier problems in our dataset, with Gemma-4-31B achieving the highest performance out of the five. We also showcase how our dataset allows us to ascertain what kind of hints help LLMs perform better at this task.

图神经网络大模型评测自动推理组合优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。