用博弈论方法拆分奖励，让大模型生成的候选结果更公平竞争。
ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Training
- 基于合作博弈的谢尔比值，将整体奖励分解为每个候选项的专属信号。
- 在多个数据集上训练更快收敛，性能优于传统方法。
- 适合需要多候选输出的推荐、编程建议等场景。
在推荐、头脑风暴和代码建议等用户-代理交互场景中，大语言模型常生成一组候选结果，目标是最大化整个集合的综合效用，而非独立评估单个候选。然而，现有强化学习后训练范式（如GRPO）通常对集合中每个候选分配相同的集合级标量奖励，导致劣质候选通过强项的高奖励获得噪声训练信号，影响探索效率。为此，本文提出谢尔比值增强型GRPO（ShapE-GRPO）。利用集合效用的排列不变性，从合作博弈论推导出一种谢尔比值增强的分解形式，将集合级奖励转化为细粒度的候选专属信号。我们证明该方法在保持谢尔比值基本公理的同时，计算复杂度为多项式时间。实验表明，ShapE-GRPO在多个数据集上均显著优于标准GRPO，且训练收敛速度加快。
原文摘要 · Abstract (English)
In user-agent interaction scenarios such as recommendation, brainstorming, and code suggestion, Large Language Models (LLMs) often generate sets of candidate recommendations where the objective is to maximize the collective utility of the entire set rather than individual candidates independently. However, existing reinforcement learning post-training paradigms, such as Group Relative Policy Optimization (GRPO), typically assign the same set-level scalar reward to every candidate in the set. This leads to noisy training signals where poor candidates free-ride on the high reward produced by a single strong peer, resulting in suboptimal exploration. To address this, we propose Shapley-Enhanced GRPO (ShapE-GRPO). By leveraging the permutation-invariant nature of set-level utility, we derive a Shapley-enhanced formulation from cooperative game theory to decompose set-level rewards into granular, candidate-specific signals. We show that our formulation preserves the fundamental axioms of the Shapley value while remaining computationally efficient with polynomial-time complexity. Empirically, ShapE-GRPO consistently outperforms standard GRPO across diverse datasets with accelerated convergence during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。