arXiv:2605.11461cs.AIcs.LG2026-05被引 1

让大模型协作解题,避免只选单一答案,提升推理多样性和准确性。

Breaking $\textit{Winner-Takes-All}$: Cooperative Policy Optimization Improves Diverse LLM Reasoning

论文配图:Breaking $\textit{Winner-Takes-All}$: Cooperative Policy Optimization Improves Diverse LLM Reasoning
图 1 · 摘自论文原文
  • 用团队贡献度替代个体竞争,奖励真正带来新解法的推理路径。
  • 在多个基准上,准确率和解法多样性均显著优于现有方法。
  • 适合需要多角度推理的复杂任务,如数学和逻辑题生成。

基于验证器的强化学习(RLVR)已成为提升大语言模型(LLM)推理能力的核心范式,但主流的组级优化算法(如GRPO)常因探索坍缩问题而陷入少数高分模式,缺乏对新解法的探索。现有方法虽引入熵正则化或多样性奖励,但未改变“赢家通吃”的竞争本质——各推理路径仍为自身优势竞争,而非协同提升整体多样性。本文提出组合作政策优化(GCPO),将训练范式从路径竞争转向团队协作:不再独立评分,而是依据单条推理对团队有效解覆盖范围的贡献进行奖励。该覆盖范围由奖励加权语义嵌入的行列式体积衡量,仅正确且非冗余的推理路径才计入。优势估计阶段,通过平均边际贡献将集体奖励重新分配给各路径。这一合作机制引导优化聚焦于非冗余的正确推理路径。在多个推理基准上的实验表明,GCPO显著提升了推理准确率与解法多样性。代码将发布于 https://github.com/bradybuddiemarch/gcpo。

原文摘要 · Abstract (English)

Reinforcement learning with verifiers (RLVR) has become a central paradigm for improving LLM reasoning, yet popular group-based optimization algorithms like GRPO often suffer from exploration collapse, where the models prematurely converge on a narrow set of high-scoring patterns, lacking the ability to explore new solutions. Recent efforts attempt to alleviate this by adding entropy regularization or diversity bonus. However, these approaches do not change the \textit{winner-takes-all} nature, where rollouts still compete for individual advantage rather than cooperating for maximizing global diversity. In this work, we propose Group Cooperative Policy Optimization (GCPO), which shifts the training paradigm from rollout competition to team cooperation. Specifically, GCPO replaces independent rollout scoring with team-level credit assignment: a rollout is rewarded by how much it contributes to the team's valid solution coverage, rather than its individual accuracy. This coverage is described as a determinant volume over reward-weighted semantic embeddings, where only correct and non-redundant rollouts contribute to this volume. During advantage estimation, GCPO redistributes the collective team reward to each single rollout according to its average marginal contribution to the team. This cooperative training paradigm routes optimization toward non-redundant correct reasoning paths. Experiments across multiple reasoning benchmarks demonstrate that GCPO significantly improves both reasoning accuracy and solution diversity over existing approaches. Code will be released at https://github.com/bradybuddiemarch/gcpo.

大模型推理强化学习多样性优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。