arXiv:2605.07147cs.LOcs.AI2026-05被引 2

用真实数据构建评测集,检验大模型能否判断数学库代码合并是否合格。

MathlibPR: Pull Request Merge-Readiness Benchmark for Formal Mathematical Libraries

  • 基于Mathlib4真实提交历史构建评测基准
  • 大模型和智能体难以区分可合并与仅通过编译的代码
  • 为自动化评审工具和奖励模型提供训练基础

Lean与Mathlib生态系统已成为大型语言模型(LLM)辅助形式化推理的行业标准,近年来取得显著成果。然而,这些成果仅将Mathlib作为依赖项使用,未直接贡献于其发展。与此同时,Mathlib的增长因评审流程瓶颈而受限——需人工评审者判断提交的拉取请求(PR)是否符合规范、值得集成。为此,我们提出MathlibPR,一个基于真实Mathlib4 PR历史构建的评测基准。我们设计分阶段评估协议,对DeepSeek、Qwen、Goedel、Kimina等大模型以及Codex、Claude Code等大模型代理进行评估。令人惊讶的是,大模型和代理均难以区分可合并的PR与仅通过编译但被修改或未合并的PR。通过将Mathlib PR历史转化为监督信号,MathlibPR为评审助手和奖励模型的发展迈出关键一步,有望帮助评估PR并引导大模型生成可合并的数学库贡献。

原文摘要 · Abstract (English)

The ecosystem of Lean and Mathlib has become the de facto standard for large language model (LLM) assisted formal reasoning with remarkable successes in recent years. Those successes, however, only consume Mathlib as an essential dependency but do not directly contribute to it. In the meantime, the growth of Mathlib has recently been bottlenecked by the review process, which requires human reviewers to judge whether proposed pull requests (PRs) follow the Mathlib's conventions and are worth integrating as part of a shared mathematical infrastructure. This leads to our central question: can LLMs help review Mathlib PRs? To this end, we introduce MathlibPR, a benchmark built from real Mathlib4 PR histories. We further propose a staged evaluation protocol and use it to evaluate both LLM models (e.g., DeepSeek, Qwen, Goedel, and Kimina) and LLM agents (e.g., Codex and Claude Code). Surprisingly, both LLM models and LLM agents struggle to distinguish merge-ready PRs from build-passing PRs that were revised or never merged. By turning Mathlib PR histories into a supervised signal, MathlibPR provides a step toward reviewer assistants and reward models that could help evaluate PRs and steer LLMs toward producing merge-ready Mathlib contributions.

形式化推理代码评审大模型评测数学库

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。