让大模型通过自我反思和强化学习实现自进化，无需真实数据也能提升表现。
Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
- 失败后生成反思文本，再尝试改进，成功则奖励反思内容。
- 数学公式生成提升34.7%，函数调用准确率提高18.1%。
- 小模型（1.5B~7B）比大模型更优，适合低资源场景。
我们探索了一种通过自我反思与强化学习提升大语言模型性能的方法。当模型任务失败时，它会生成一段自我反思的评述，分析先前错误；随后在反思提示下再次尝试任务。若第二次成功，则对反思阶段生成的词元进行奖励。实验表明，该框架在多种模型架构上均带来显著性能提升：数学方程写作任务最高提升34.7%，函数调用任务提升18.1%。值得注意的是，参数量为15亿至70亿的小型微调模型，表现优于同系列十倍大的模型。该方法在无法生成合成数据、仅能获取二元反馈的条件下，仍能有效推动模型自提升，是构建更可靠语言模型的重要路径。
原文摘要 · Abstract (English)
We explore a method for improving the performance of large language models through self-reflection and reinforcement learning. By incentivizing the model to generate better self-reflections when it answers incorrectly, we demonstrate that a model's ability to solve complex, verifiable tasks can be enhanced even when generating synthetic data is infeasible and only binary feedback is available. Our framework operates in two stages: first, upon failing a given task, the model generates a self-reflective commentary analyzing its previous attempt; second, the model is given another attempt at the task with the self-reflection in context. If the subsequent attempt succeeds, the tokens generated during the self-reflection phase are rewarded. Our experimental results show substantial performance gains across a variety of model architectures, as high as 34.7% improvement at math equation writing and 18.1% improvement at function calling. Notably, smaller fine-tuned models (1.5 billion to 7 billion parameters) outperform models in the same family that are 10 times larger. Our novel paradigm is thus an exciting pathway to more useful and reliable language models that can self-improve on challenging tasks with limited external feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。