让大模型自己评估答案好坏，用自反馈强化学习提升推理能力。
Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
- 用模型自身对答案的置信度生成内在奖励信号
- 在数学推理和多选题上准确率显著提升，概率校准更优
- 无需人工标注，适合想提升模型推理可信度的研究者
大型语言模型常产生看似合理但校准不足的答案，限制其在需要推理的任务中的可靠性。我们提出基于自反馈的强化学习（RLSF），一个后训练阶段，利用模型自身的置信度作为内在奖励，模仿人类在无外部反馈下的学习方式。在冻结的LLM生成多个思维链解法后，我们定义并计算每个最终答案片段的置信度，并据此对生成轨迹进行排序。这些合成偏好随后用于标准偏好优化微调策略，类似于RLHF，但无需人工标注、黄金答案或外部奖励。RLSF同时实现（i）改进模型的概率估计，恢复良好的校准性；（ii）增强逐步推理能力，在算术推理和多项选择问答任务上表现更优。通过将模型自身的不确定性转化为有效自反馈，RLSF验证了内在行为驱动的强化学习作为大模型后训练流程中一种原则性强且数据高效的组成部分，值得进一步探索内在奖励在后训练中的应用。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often produce plausible but poorly-calibrated answers, limiting their reliability on reasoning-intensive tasks. We present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that uses the model's own confidence as an intrinsic reward, mimicking how humans learn in the absence of external feedback. After a frozen LLM generates several chain-of-thought solutions, we define and compute the confidence of each final answer span and rank the traces accordingly. These synthetic preferences are then used to fine-tune the policy with standard preference optimization, similar to RLHF yet requiring no human labels, gold answers, or externally curated rewards. RLSF simultaneously (i) refines the model's probability estimates -- restoring well-behaved calibration -- and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering. By turning a model's own uncertainty into useful self-feedback, RLSF affirms reinforcement learning on intrinsic model behaviour as a principled and data-efficient component of the LLM post-training pipeline and warrents further research in intrinsic rewards for LLM post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。