用模型自信心做奖励，少样本微调让大模型解题更准
Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models
- 用模型自身置信度当奖励信号，无需人工标注或外部奖励
- 仅用16样本+10~20步训练，数学题准确率提升9.7%~21.7%
- 适合需要低成本、无监督微调的推理类大模型应用
大型语言模型在推理方面表现优异，但后训练对对齐其行为与任务目标仍至关重要。现有强化学习方法常依赖昂贵的人工标注或外部奖励模型。我们提出基于自信心的强化学习（RLSC），利用模型自身的置信度作为奖励信号，消除对标签、偏好模型或奖励工程的需求。在Qwen2.5-Math-7B上，每道题仅使用16个样本，经过10或20步训练，RLSC在AIME2024上准确率提升13.4%，MATH500提升21.2%，Minerva Math提升21.7%，Olympiadbench提升20.8%，AMC23提升9.7%。RLSC提供了一种简单、可扩展的推理模型后训练方法，仅需少量样本和无标注监督。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at reasoning, yet post-training remains critical for aligning their behavior with task goals. Existing reinforcement learning (RL) methods often depend on costly human annotations or external reward models. We propose Reinforcement Learning via Self-Confidence (RLSC), which uses the model's own confidence as reward signals-eliminating the need for labels, preference models, or reward engineering. Applied to Qwen2.5-Math-7B with only 16 samples per question and 10 or 20 training steps, RLSC improves accuracy by +13.4% on AIME2024, +21.2% on MATH500, +21.7% on Minerva Math, +20.8% on Olympiadbench, and +9.7% on AMC23. RLSC provides a simple, scalable post-training method for inference models, requiring only a small number of samples and unlabelled supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。