arXiv:2507.20252cs.CLcs.AI2025-07被引 2

让语言模型在生成结束后继续自我评估,提升推理与自检能力。

Post-Completion Learning for Language Models

  • 模型生成后继续输出自评内容,利用后完成空间进行训练
  • 在多个数据集上显著优于传统SFT和RL方法,提升输出质量
  • 兼顾推理与评估能力,适合追求高质量生成的场景

当前语言模型训练通常在遇到结束符(<eos>)时终止,忽略了生成完成后仍存在的学习机会。本文提出后完成学习(Post-Completion Learning, PCL),一种新型训练框架,系统利用模型输出完成后的序列空间,以增强推理与自评估能力。PCL使模型在训练中持续生成自我评估和奖励预测,而推理阶段仍可在完成点停止,保持高效。为此,我们设计了一种白盒强化学习方法:模型根据奖励规则评估输出内容,计算得分并与奖励函数对齐以实现监督。通过双轨监督微调(SFT)优化推理与评估能力,并与强化学习结合,实现多目标混合优化。实验结果表明,在不同数据集和模型上,PCL均持续优于传统SFT与强化学习方法。该方法为语言模型训练提供了新路径,在不牺牲部署效率的前提下提升输出质量。

原文摘要 · Abstract (English)

Current language model training paradigms typically terminate learning upon reaching the end-of-sequence (<eos>) token, overlooking the potential learning opportunities in the post-completion space. We propose Post-Completion Learning (PCL), a novel training framework that systematically utilizes the sequence space after model output completion, to enhance both the reasoning and self-evaluation abilities. PCL enables models to continue generating self-assessments and reward predictions during training, while maintaining efficient inference by stopping at the completion point. To fully utilize this post-completion space, we design a white-box reinforcement learning method: let the model evaluate the output content according to the reward rules, then calculate and align the score with the reward functions for supervision. We implement dual-track SFT to optimize both reasoning and evaluation capabilities, and mixed it with RL training to achieve multi-objective hybrid optimization. Experimental results on different datasets and models demonstrate consistent improvements over traditional SFT and RL methods. Our method provides a new technical path for language model training that enhances output quality while preserving deployment efficiency.

语言模型自评估强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。