pass@k评估虽有用，但不适合作为强化学习的优化目标
Pass@k Metric for RLVR: A Diagnostic Tool of Exploration, But Not an Objective
- 将pass@k视为对pass@1的逐样本正例重加权机制
- 在探索最关键的阶段，pass@k学习信号几乎消失
- 适合诊断模型探索能力，但不适合直接用于训练
大型语言模型（LLMs）执行复杂多步推理的能力是现代人工智能研究的核心。为评估和提升该能力，通过在k次独立采样中至少获得一次正确解的概率来衡量的pass@k指标受到广泛关注。其直观性使其不仅被用作评估标准，还被当作强化学习中的直接优化目标。本文分析了pass@k目标，推导其梯度，揭示其本质是对更简单的pass@1目标进行逐样本正例重加权。分析表明，在探索最关键的区域，pass@k提供的学习信号趋于消失。我们进一步分析了“探索崩溃”动态，发现当策略集中概率质量时，pass@k与pass@1之间的差距缩小。结论是：虽然pass@k是有效的诊断工具，但可能不适合作为直接优化目标。相反，显式鼓励高效探索的机制或能为推理任务中的强化学习提供更有效路径。
原文摘要 · Abstract (English)
The ability of Large Language Models (LLMs) to perform complex, multi-step reasoning is a central focus of modern AI research. To evaluate and enhance this capability, the pass@k metric, which measures the probability of obtaining at least one correct solution in k independent samples, has received significant attention. Its intuitive appeal has led to its adoption not only as an evaluation standard but also as a direct optimization objective in reinforcement learning. In this paper, we analyze the pass@k objective, derive its gradient, and demonstrate that it is fundamentally a per-example positive reweighting of the simpler pass@1 objective. Our analysis reveals that the pass@k objective provides a vanishing learning signal in regimes where exploration is most critical. We further analyze the dynamics of "exploration collapse", showing that as the policy concentrates probability mass, the gap between pass@k and pass@1 diminishes. We conclude that while pass@k is a useful diagnostic tool, it may be an unsuitable direct objective for optimization. Instead, mechanisms explicitly encouraging efficient exploration could offer a more effective path forward for reinforcement learning in reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。