优化Pass@k会降低Pass@1,因提示干扰导致梯度冲突。
Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-training
- 通过分析提示干扰,揭示多样本优化的梯度冲突机制。
- 实验证明,提升Pass@k时Pass@1常下降,尤其在数学推理任务中。
- 适合关注推理效率与单次生成质量的研究者参考。
Pass@k 是衡量可验证大语言模型任务性能的常用指标,包括数学推理、代码生成和简答推理。该指标定义为:在独立采样的 k 个解中,只要有一个通过验证器即视为成功。这一多样本推理指标推动了直接优化 Pass@k 的推理感知微调方法。然而,先前研究发现一个反复出现的权衡:在优化 Pass@k 时,Pass@1 常出现下降。这一权衡具有实际意义,因为受延迟与成本预算限制、验证器覆盖不全以及对可靠单次回退的需求,Pass@1 往往仍面临严格约束。本文研究该权衡的根源,提供了关于 Pass@k 策略优化如何通过提示干扰引发梯度冲突从而降低 Pass@1 的理论刻画。我们发现,Pass@k 的策略梯度可能与 Pass@1 梯度冲突,因为 Pass@k 优化隐式地将提示重加权至低成功率提示;当这些提示属于负向干扰类型时,其被抬高会将 Pass@k 更新方向从 Pass@1 方向偏移。我们在可验证数学推理任务的大语言模型上验证了这一理论发现。
原文摘要 · Abstract (English)
Pass@k is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning. It defines success if any of $k$ independently sampled solutions passes a verifier. This multi-sample inference metric has motivated inference-aware fine-tuning methods that directly optimize pass@$k$. However, prior work reports a recurring trade-off: pass@k improves while pass@1 degrades under such methods. This trade-off is practically important because pass@1 often remains a hard operational constraint due to latency and cost budgets, imperfect verifier coverage, and the need for a reliable single-shot fallback. We study the origin of this trade-off and provide a theoretical characterization of when pass@k policy optimization can reduce pass@1 through gradient conflict induced by prompt interference. We show that pass@$k$ policy gradients can conflict with pass@1 gradients because pass@$k$ optimization implicitly reweights prompts toward low-success prompts; when these prompts are what we term negatively interfering, their upweighting can rotate the pass@k update direction away from the pass@1 direction. We illustrate our theoretical findings with large language model experiments on verifiable mathematical reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。