让语言模型在生成结束后继续自我评估,提升推理与自检能力。
Post-Completion Learning for Language Models
- 模型生成后继续输出自评内容,利用后完成空间进行训练
- 在多个数据集上显著优于传统SFT和RL方法,提升输出质量
- 兼顾推理与评估能力,适合追求高质量生成的场景
当前语言模型训练通常在遇到结束符(<eos>)时终止,忽略了生成完成后仍存在的学习机会。本文提出后完成学习(Post-Completion Learning, PCL),一种新型训练框架,系统利用模型输出完成后的序列空间,以增强推理与自评估能力。PCL使模型在训练中持续生成自我评估和奖励预测,而推理阶段仍可在完成点停止,保持高效。为此,我们设计了一种白盒强化学习方法:模型根据奖励规则评估输出内容,计算得分并与奖励函数对齐以实现监督。通过双轨监督微调(SFT)优化推理与评估能力,并与强化学习结合,实现多目标混合优化。实验结果表明,在不同数据集和模型上,PCL均持续优于传统SFT与强化学习方法。该方法为语言模型训练提供了新路径,在不牺牲部署效率的前提下提升输出质量。
原文摘要 · Abstract (English)
Current language model training paradigms typically terminate learning upon reaching the end-of-sequence (<eos>) token, overlooking the potential learning opportunities in the post-completion space. We propose Post-Completion Learning (PCL), a novel training framework that systematically utilizes the sequence space after model output completion, to enhance both the reasoning and self-evaluation abilities. PCL enables models to continue generating self-assessments and reward predictions during training, while maintaining efficient inference by stopping at the completion point. To fully utilize this post-completion space, we design a white-box reinforcement learning method: let the model evaluate the output content according to the reward rules, then calculate and align the score with the reward functions for supervision. We implement dual-track SFT to optimize both reasoning and evaluation capabilities, and mixed it with RL training to achieve multi-objective hybrid optimization. Experimental results on different datasets and models demonstrate consistent improvements over traditional SFT and RL methods. Our method provides a new technical path for language model training that enhances output quality while preserving deployment efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。