提出新评估指标Cover@tau,更准确衡量大模型真实推理能力。
Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries
- 用Cover@tau替代Pass@k,以可靠比例衡量推理能力
- 发现基线模型在大量采样下表现更好是因随机猜测而非真推理
- 适合关注模型可靠性与真推理能力的研究者使用
强化学习结合可验证奖励(RLVR)已成为提升大语言模型在编程、数学、逻辑等推理任务表现的强大范式。当前研究通常通过大采样预算下的Pass@k评估模型的推理边界(即能解决的问题比例)。近期研究表明存在交叉现象:小k值时RLVR模型优于基线模型,但当采样数量极大时,基线模型反而表现更优,这被解释为基线模型具有更大的推理边界。我们指出,在离散答案空间任务(如数值输出的数学题)中,大k值下的Pass@k反映的是试验次数趋近无穷时成功概率的增加,而非真正的推理能力,因此可能误导判断。为此,我们提出Cover@tau指标,衡量在至少tau比例完成结果正确的前提下,模型能解决的问题比例。不同于Pass@k,Cover@tau显式引入可靠性阈值:依赖随机猜测的模型在tau增大时性能迅速下降。我们在多个RLVR模型上评估了基于Cover@tau的指标,揭示了不同算法相对排名与Pass@1的不同,提供了理解推理边界的新视角。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm to improve Large Language Models on reasoning tasks such as coding, math or logic. To assess the reasoning boundary (the fraction of problems a model can solve) researchers often report Pass@k at large sampling budgets. Recent results reveal a crossover phenomenon: while RLVR models outperform the base model at small k values, the base model usually outperforms them when sampling a very large number of completions. This has been interpreted as evidence that base models have a larger reasoning boundary. We argue that on tasks with discrete answer spaces, such as math with numeric outputs, Pass@k at large k reflects the increasingly higher chance of success in the limit of the number of trials rather than genuine reasoning, and can therefore be misleading. We propose Cover@tau, which measures the fraction of problems that a model can solve for which at least a tau proportion of completions are correct. Unlike Pass@k, Cover@tau captures reasoning under an explicit reliability threshold: models that rely on random guessing degrade rapidly as tau increases. We evaluate several RLVR models using Cover@tau-based metrics and illustrate how the relative rankings of popular algorithms change compared to Pass@1, offering a different perspective on reasoning boundaries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。