arXiv:2602.03061cs.LGcs.AI2026-02被引 2

利用模型对推理链的比较信号,提升大模型数学能力评估的准确性。

Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals

  • 用模型自身判断推理链优劣,作为辅助评估信号
  • 在小样本下显著降低评估方差,提升排名稳定性
  • 适合需要精准比较模型性能的研究者使用

大语言模型在数学推理评估中受限于基准数据集规模小和模型随机性,导致准确率估计方差高、跨平台排名不稳定。在难题上,模型虽无法给出正确最终答案,但仍可提供可靠的两两推理链比较信号。本文基于此设计了一种统计高效的评估框架,将标准标注结果与模型生成的辅助推理链比较信号结合,将比较信号视为控制变量,构建基于有效影响函数(EIF)的半参数估计器。该一阶估计器达到半参数效率界,严格优于朴素平均法,且具备渐近正态性,支持可靠置信区间计算。仿真显示,随着模型输出噪声增加,排名准确性显著提升。在GPQA Diamond、AIME 2025和GSM8K上的实验进一步证明,该方法在小样本条件下能实现更精确的性能估计和更可靠的模型排序。

原文摘要 · Abstract (English)

Evaluating mathematical reasoning in LLMs is constrained by limited benchmark sizes and inherent model stochasticity, yielding high-variance accuracy estimates and unstable rankings across platforms. On difficult problems, an LLM may fail to produce a correct final answer, yet still provide reliable pairwise comparison signals indicating which of two candidate solutions is better. We leverage this observation to design a statistically efficient evaluation framework that combines standard labeled outcomes with pairwise comparison signals obtained by having models judge auxiliary reasoning chains. Treating these comparison signals as control variates, we develop a semiparametric estimator based on the efficient influence function (EIF) for the setting where auxiliary reasoning chains are observed. This yields a one-step estimator that achieves the semiparametric efficiency bound, guarantees strict variance reduction over naive sample averaging, and admits asymptotic normality for principled uncertainty quantification. Across simulations, our one-step estimator substantially improves ranking accuracy, with gains increasing as model output noise grows. Experiments on GPQA Diamond, AIME 2025, and GSM8K further demonstrate more precise performance estimation and more reliable model rankings, especially in small-sample regimes where conventional evaluation is pretty unstable.

数学推理评估方法大模型评价统计推断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。