用真实科研难题测试大模型深度推理能力,发现顶尖模型表现极差。
FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming
- 基于一阶逻辑构造复杂图论问题,模拟真实科研挑战。
- 顶级模型o3在10次尝试下正确率不足1%,暴露推理短板。
- 适合研究模型深层理解、理论计算机科学与自动问题生成者。
前沿大模型虽知识广博,但距离真正专家水平仍有巨大差距。为揭示其真实能力边界,本文放弃人为设计的编程竞赛题,转而聚焦真实科研问题。我们构建了FormulaOne基准,融合图论、逻辑与算法,均在大模型训练分布内。该数据集具备三大特性:一是具有商业价值,关联路由、调度与网络设计等大规模优化问题;二是基于高表达力的单量二阶(MSO)逻辑生成,支持规模化自动建模,适用于强化学习环境;三是与理论计算机科学前沿紧密相关,涉及强指数时间假设(SETH)等核心猜想。任何在该数据集上超越已有结果的进展,都可能带来深远理论意义。令人震惊的是,即使提供10次尝试和少量示例,OpenAI o3模型正确率仍低于1%,凸显其在部分领域距专家级理解仍有显著差距。为此,我们还推出了FormulaOne-Warmup,提供同分布下的简化任务。全部数据集与评估框架已公开。
原文摘要 · Abstract (English)
Frontier AI models demonstrate formidable breadth of knowledge. But how close are they to true human -- or superhuman -- expertise? Genuine experts can tackle the hardest problems and push the boundaries of scientific understanding. To illuminate the limits of frontier model capabilities, we turn away from contrived competitive programming puzzles, and instead focus on real-life research problems. We construct FormulaOne, a benchmark that lies at the intersection of graph theory, logic, and algorithms, all well within the training distribution of frontier models. Our problems are incredibly demanding, requiring an array of reasoning steps. The dataset has three key properties. First, it is of commercial interest and relates to practical large-scale optimisation problems, such as those arising in routing, scheduling, and network design. Second, it is generated from the highly expressive framework of Monadic Second-Order (MSO) logic on graphs, paving the way toward automatic problem generation at scale; ideal for building RL environments. Third, many of our problems are intimately related to the frontier of theoretical computer science, and to central conjectures therein, such as the Strong Exponential Time Hypothesis (SETH). As such, any significant algorithmic progress on our dataset, beyond known results, could carry profound theoretical implications. Remarkably, state-of-the-art models like OpenAI's o3 fail entirely on FormulaOne, solving less than 1% of the questions, even when given 10 attempts and explanatory fewshot examples -- highlighting how far they remain from expert-level understanding in some domains. To support further research, we additionally curate FormulaOne-Warmup, offering a set of simpler tasks, from the same distribution. We release the full corpus along with a comprehensive evaluation framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。