RAPID让小模型强化学习训练提速11%-34%，兼顾效率与精度。
RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models
- 批量推理+小批次离策略更新，优化计算资源利用
- 实验显示在三个基准上提速11%-34%，精度相当或更优
- 适合追求高效微调的小模型应用，如数学与编码任务
强化学习（RL）已成为微调小语言模型（SLMs）以解决数学和编程等特定任务的有前景方法。然而，传统RL算法通常资源消耗大，训练耗时长。本文提出RAPID，一种新型强化学习算法，可显著缩短训练时间。核心思路是：RL耗时主要源于训练中需同时进行推理与反向传播。为此，算法采用大批次推理，再以小批次进行离策略策略梯度更新。针对离策略学习带来的偏差，引入分组优势估计，并推导出重要性加权估计器进行修正。实验表明，相比现有最优RL算法，RAPID在三个基准测试上将运行时间减少11%-34%，同时保持相似或更优的准确率。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as a promising strategy for finetuning small language models (SLMs) to solve targeted tasks such as math and coding. However, RL algorithms tend to be resource-intensive, taking a significant amount of time to train. We propose RAPID, a novel RL algorithm that can substantially reduce the running time of RL. Our key insight is that RL tends to be costly due to the need to perform both inference and backpropagation during training. To maximize use of computational resources, our algorithm performs inference in large batches, and then performs off-policy policy gradient updates in mini-batches. For off-policy updates, we incorporate group advantage estimation into the policy gradient algorithm, and derive an importance weighted estimator to correct for the bias arising from off-policy learning. Our experiments demonstrate that our algorithm can reduce running time by 11%-34% on three benchmarks compared to state-of-the-art RL algorithms while maintaining similar or better accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。