用强化学习优化指令遵循，比传统微调更有效
RLSR: Reinforcement Learning with Supervised Reward Outperforms SFT in Instruction Following
- 用语义相似度作为奖励，让模型生成更贴近人类回答
- 在Qwen-7B上指令跟随胜率提升至26.34%，优于SFT的21.01%
- 可替代或配合SFT，适合追求高精度指令理解的场景
大语言模型预训练后，常采用SFT、RLHF、RLVR和RFT等技术提升指令遵循能力、减少不良响应、增强推理能力并实现小样本领域适配。SFT依赖下一个词预测目标，利用大量人工标注回复数据增强基础模型的指令遵循能力。相比之下，RFT采用基于强化学习的方法，在有限监督下将微调后的推理模型适配到特定领域。受RFT启发，我们提出用RLSR替代SFT，以在强化学习框架中充分利用大规模SFT数据，进一步提升基础模型的指令遵循能力。在RLSR中，基础模型为每个提示生成多个回复，奖励分数通过生成回复与人工标注回复在语义嵌入空间中的余弦相似度计算。RLSR可直接替代SFT，显著提升指令遵循基准表现——例如，使用RLSR(SB)的Qwen-7B(INFINITY)在AlpacaEval上的胜率为26.34%，超过SFT的21.01%。此外，结合SFT与RLSR能进一步提升下游任务表现，该组合在相同模型上达到30.73%的胜率。
原文摘要 · Abstract (English)
After the pretraining stage of LLMs, techniques such as SFT, RLHF, RLVR, and RFT are applied to enhance instruction-following ability, mitigate undesired responses, improve reasoning capability and enable efficient domain adaptation with minimal data. SFT relies on the next-token prediction objective to strengthen instruction following in a base model using a large corpus of human-labeled responses. In contrast, RFT employs a RL-based approach to adapt fine-tuned reasoning models to specific domains with limited supervision. Inspired by RFT, we propose replacing SFT with RLSR to leverage the extensive SFT dataset in an RL framework, thereby improving the base model's instruction-following ability. In RLSR, the base model generates multiple responses for each prompt, and reward scores are computed as the cosine similarity in the semantic embedding space between the generated and human-labeled responses. RLSR can be utilized in multiple ways. It can directly replace SFT, achieving superior performance on instruction-following benchmarks-for example, RLSR (SB) on Qwen-7B (INFINITY) achieved an AlpacaEval win rate of 26.34%, surpassing SFT's 21.01%. Furthermore, combining SFT and RLSR further enhances downstream task performance; Qwen-7B (INFINITY) achieved a win rate of 30.73% when trained with SFT + RLSR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。