让大模型训练更符合实际评估指标，提升成功率与最高分表现。
RSPO: Risk-Seeking Policy Optimization for Pass@k and Max@k Metrics in Large Language Models
- 直接优化 Pass@k 与 Max@k 指标，解决风险偏好不匹配问题。
- 提出闭式概率公式缓解共现干扰，避免低分响应被错误强化。
- 理论严谨且实验充分，适合追求高可靠性输出的场景使用。
当前大模型后训练采用风险中性目标以最大化期望奖励，但评估却依赖风险偏好型指标如 Pass@k（k 次生成中至少一次成功）和 Max@k（k 次响应中的最大奖励）。这种风险偏好不一致必然导致性能不佳。为此，我们提出风险寻求策略优化（RSPO），在训练中直接针对 Pass@k 与 Max@k 进行优化。优化这些指标的关键挑战在于“搭便车”问题：若某次生成中存在高奖励响应，即使其他响应得分低，也会因共现被意外强化，造成优化效率下降。RSPO 通过利用给定响应在 k 次采样中为最大值的闭式概率，解决了这一难题。尽管涉及多响应嵌套梯度，RSPO 仍能生成高效且无偏的梯度估计器。我们通过严格的理论分析和全面的实验验证了方法的有效性。
原文摘要 · Abstract (English)
Current large language model post-training optimizes a risk-neutral objective that maximizes expected reward, yet evaluation relies heavily on risk-seeking metrics like Pass@k (at least one success in k trials) and Max@k (maximum reward across k responses). This mismatch in risk preferences can inevitably lead to suboptimal performance. To bridge this gap, we propose Risk-Seeking Policy Optimization (RSPO), a novel method that directly targets Pass@k and Max@k during training. A key challenge in optimizing these metrics is the "hitchhiking" problem: low-reward responses are inadvertently reinforced if they co-occur with a high-reward response within a sample of k generations, resulting in inefficient optimization. RSPO addresses this problem by leveraging the closed-form probability that a given response is the maximum among k samplings. Despite the complexity of nested gradients over multiple responses, RSPO produces efficient, unbiased gradient estimators for both metrics. We validate our approach with both rigorous theoretical analysis and comprehensive experimental results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。