小模型通过自我迭代反馈学习推理，无需外部标注。
Learning to Reason via Self-Iterative Process Feedback for Small Language Models
- 用自生成正负样本进行偏好优化，实现自我反馈训练。
- 在GSM8K上准确率提升12.43，在MBPP上通过率提升3.95。
- 适合需要高效推理的小模型应用，如资源受限场景。
小语言模型（SLMs）比大语言模型更高效、成本更低且可定制，但在推理等特定任务上表现较差。以往提升SLM推理能力的方法如监督微调和知识蒸馏，常依赖昂贵的外部信号，导致模型在监督不足时过度自信，能力受限。为此，本文提出让SLMs通过自迭代反馈学习推理，结合几率比偏好优化（ORPO），利用模型自身生成的正负样本进行微调与对齐。同时，引入基于采样的推理模拟和过程奖励模型，实现推理过程的奖励监督。相较于监督微调（SFT），该方法使Gemma-2B在GSM8K上的准确率提升12.43，在MBPP上的Pass@1提升3.95。此外，在MMLU_Math和HumanEval等跨域任务上也展现出更强的泛化能力。
原文摘要 · Abstract (English)
Small language models (SLMs) are more efficient, cost-effective, and customizable than large language models (LLMs), though they often underperform in specific areas like reasoning. Past methods for enhancing SLMs' reasoning, such as supervised fine-tuning and distillation, often depend on costly external signals, resulting in SLMs being overly confident with limited supervision signals, thus limiting their abilities. Therefore, this study enables SLMs to learn to reason from self-iterative feedback. By combining odds ratio preference optimization (ORPO), we fine-tune and align SLMs using positive and negative signals generated by themselves. Additionally, we introduce process supervision for rewards in preference alignment by sampling-based inference simulation and process reward models. Compared to Supervised Fine-Tuning (SFT), our method improves the performance of Gemma-2B by 12.43 (Acc) on GSM8K and 3.95 (Pass@1) on MBPP. Furthermore, the proposed method also demonstrated superior out-of-domain generalization capabilities on MMLU_Math and HumanEval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。