大模型仅靠自身生成的答案就能自我提升推理能力。
A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
- 用模型自己生成问题和答案，循环训练以提升推理。
- 在六个数学推理数据集上，优于未微调的基线模型。
- 在线更新数据与温度动态是关键，无需外部奖励。
语言模型能否仅通过自身采样响应进行训练，而无需外部奖励来提升推理性能？我们证明这是可行的。本文提出自进化后训练（SePT），一种简单的后训练方法：在自我生成与基于自生成数据的训练之间交替进行。该方法反复采样问题，使用模型自身在指定采样温度下生成回答，并以这些自生成数据训练模型。在自训练循环中，采用在线数据刷新机制，每批新数据由最新更新的模型生成。在六个数学推理基准测试中，SePT提升了强基线表现（即未微调的基模型在最佳采样温度下的表现），适用于多个模型。额外消融实验表明在线数据刷新和温度动态的重要性。总体而言，我们的结果揭示了一个仅用自生成监督即可改进推理的实用范式。代码已开源：https://github.com/ElementQiii/SePT。
原文摘要 · Abstract (English)
Can language models improve their reasoning performance without external rewards, using only their own sampled responses for training? We show that they can. We propose Self-evolving Post-Training (SePT), a simple post-training method that alternates between self-generation and training on self-generated responses. It repeatedly samples questions, uses the model itself to generate responses under a specified sampling temperature, and then trains the model on the self-generated data. In this self-training loop, we use an online data refresh mechanism, where each new batch is generated by the most recently updated model. Across six math reasoning benchmarks, SePT improves a strong no-training baseline, defined as the untuned base model evaluated at its best swept decoding temperature, on several tested models. Additional ablations demonstrate the importance of online data refresh and temperature dynamics. Overall, our results identify a practical regime where reasoning can be improved using self-generated supervision alone. Our code is available at https://github.com/ElementQiii/SePT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。