用外部参考答案提升小模型推理能力，解决对比学习失效问题
GCPO: When Contrast Fails, Go Gold
- 引入外部参考答案作为标准，弥补模型自生成响应的局限
- 训练效率显著提升，所有样本均被有效利用
- 适合需要增强推理泛化的小模型研究者使用
强化学习已被广泛用于提升大语言模型的推理能力。如何拓展小模型的推理上限成为研究热点。然而，如组相对策略优化（GRPO）等算法存在明显缺陷：模型生成结果的上限完全由自身决定，无法从全错或全对的样本中获取知识。本文提出组对比策略优化（GCPO），引入外部标准参考答案。当模型无法解题时，参考答案提供正确响应，引导模型获得明确的更新方向。该方法具有两大优势：(1) 全面利用每个样本，提升训练效率；(2) 使模型在训练中模仿参考答案的解题策略，增强推理泛化能力。GCPO在多个基准数据集上表现优异，显著超越基线模型。代码已开源：https://github.com/AchoWu/GCPO。
原文摘要 · Abstract (English)
Reinforcement learning has been widely applied to enhance the reasoning capabilities of large language models. Extending the inference limits of smaller models has become a prominent research focus. However, algorithms such as Group Relative Policy Optimization (GRPO) suffer from a clear drawback: the upper bound of a model's rollout responses is entirely determined by the model itself, preventing the acquisition of knowledge from samples that are either all incorrect or all correct. In this paper, we introduce Group Contrastive Policy Optimization (GCPO), a method that incorporates external standard reference answers. When the model cannot solve a problem, the reference answer supplies the correct response, steering the model toward an unequivocally accurate update direction. This approach offers two main advantages: (1) it improves training efficiency by fully utilizing every sample; (2) it enables the model to emulate the problem solving strategy of the reference answer during training, thereby enhancing generalization in reasoning. GCPO achieves outstanding results across multiple benchmark datasets, yielding substantial improvements over the baseline model. Our code is available at: https://github.com/AchoWu/GCPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。