用对数平均融合SFT与可训练策略，提升大模型推理能力
Complementing reinforcement learning with SFT through logit averaging in the post training of LLMs
- 通过平均参考策略与可训练策略的对数几率，实现策略融合
- 在MATH、cn-k12、MMLU上准确率优于或等同于传统KL正则化方法
- 无需KL正则或评判器，更简洁且保留SFT格式优势
我们提出一种新方法：在大模型后训练中，对冻结的参考策略（如SFT）和可训练策略的对数几率进行平均，并融入分组相对策略优化（GRPO）。与需要KL正则化或评判器的可验证奖励强化学习（RLVR）方法不同，本方法通过对数几率平均结构将可训练策略与参考策略耦合，既利用了可训练策略的推理能力，又保持了SFT的格式优势。在MATH、cn-k12和MMLU三个数据集上的实验表明，该方法在准确率上高于或至少等同于标准的KL正则化GRPO。
原文摘要 · Abstract (English)
We introduce a novel method that averages the logits of a frozen reference policy (e.g., SFT) and a trainable policy, and incorporate the method into Group Relative Policy Optimization (GRPO). In contrast to Reinforcement Learning with Verifiable Rewards (RLVR) methods, our proposal does not involve a Kullback Leibler (KL) regularization or critic; the trainable policy and the reference anchor are coupled through the logit averaging structure to leverage the reasoning expertise of the trainable policy while maintaining the formatting advantage of SFT. Our method is evaluated on MATH, cn-k12, and MMLU, and the results show a higher accuracy or at least comparable accuracy relative to the canonical KL-regularized GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。