发现大模型评数学证明存在系统性偏差,最高虚高0.36分。
QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
- 构建双评分标准基准,对比课程评分与专家共识
- 前沿模型在离散数学中评分虚高,最高超人类0.36分
- 适合关注AI评估可靠性、数学推理的学者和开发者
随着大语言模型在基础评测中趋于饱和,研究前沿已从生成转向自动化评估的可靠性。我们发现,标准的‘大模型作裁判’方法在本科高年级至早期研究生水平的数学证明评估中存在系统性对齐偏差。为此,我们提出QEDBench,首个大规模双评分标准对齐基准,通过对比课程特定评分标准与专家通用知识标准,系统量化大模型与人类专家之间的对齐程度。基于7位评委×5个求解器的双重评估矩阵,覆盖1000+小时人工评估数据,结果显示部分前沿评估模型如Claude Opus 4.5、DeepSeek-V3、Qwen 2.5 Max和Llama 4 Maverick均存在显著正向偏差(平均分虚高分别为+0.18、+0.20、+0.30、+0.36)。此外,我们发现离散领域存在关键推理差距:尽管Gemini 3.0 Pro在整体表现达最优(平均人类评估得分0.91),其他推理模型如GPT-5 Pro和Claude Sonnet 4.5在离散数学中的表现显著下降,平均得分分别降至0.72和0.63;在图论中则进一步降至0.74和0.50。我们同时公开发布QEDBench,供评估与改进AI裁判使用,项目地址:https://github.com/qqliu/Yale-QEDBench。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) saturate elementary benchmarks, the research frontier has shifted from generation to the reliability of automated evaluation. We demonstrate that standard "LLM-as-a-Judge" protocols suffer from a systematic Alignment Gap when applied to upper-undergraduate to early graduate level mathematics. To quantify this, we introduce QEDBench, the first large-scale dual-rubric alignment benchmark to systematically measure alignment with human experts on university-level math proofs by contrasting course-specific rubrics against expert common knowledge criteria. By deploying a dual-evaluation matrix (7 judges x 5 solvers) against 1,000+ hours of human evaluation, we reveal that certain frontier evaluators like Claude Opus 4.5, DeepSeek-V3, Qwen 2.5 Max, and Llama 4 Maverick exhibit significant positive bias (up to +0.18, +0.20, +0.30, +0.36 mean score inflation, respectively). Furthermore, we uncover a critical reasoning gap in the discrete domain: while Gemini 3.0 Pro achieves state-of-the-art performance (0.91 average human evaluation score), other reasoning models like GPT-5 Pro and Claude Sonnet 4.5 see their performance significantly degrade in discrete domains. Specifically, their average human evaluation scores drop to 0.72 and 0.63 in Discrete Math, and to 0.74 and 0.50 in Graph Theory. In addition to these research results, we also release QEDBench as a public benchmark for evaluating and improving AI judges. Our benchmark is publicly published at https://github.com/qqliu/Yale-QEDBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。