用强化学习提升小模型数学推理效率，省40% token还更准
Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning
- 用强化学习优化预算强制策略，让小模型更省计算资源
- 仅用1500样本训练，数学推理准确率更高，token消耗降超40%
- 适合追求高效推理的小模型部署，尤其擅长数学题
测试时扩展方法因计算高效且无需参数训练而广受欢迎，用于提升大语言模型的推理能力。其中预算强制是一种解码干预策略，通过分配额外计算资源促进模型自我修正。但该方法依赖长上下文推理轨迹的监督微调，导致小模型因输出冗长而性能下降。为此，我们提出融合强化学习的框架，在仅使用1500个训练样本的情况下，显著提升了1.5B模型在数学推理任务上的表现。实验表明，该SFT+RL模型在不同计算预算下均优于纯SFT模型，整体准确率更高，同时token使用量减少40%以上，证明强化学习可有效缓解长上下文训练带来的损失，全面增强数学推理能力。
原文摘要 · Abstract (English)
Test-time scaling methods have seen a rapid increase in popularity for its computational efficiency and parameter-independent training to improve reasoning performance on Large Language Models. One such method is called budget forcing, a decoding intervention strategy which allocates extra compute budget for thinking and elicits the inherent self-correcting behavior of the model. However, this relies on supervised fine-tuning (SFT) on long-context reasoning traces which causes performance degradation on smaller models due to verbose responses. For this reason, we offer a framework integrating reinforcement learning (RL) to improve token efficiency and boost the performance of a 1.5B model for mathematical reasoning. We demonstrate this using only 1.5K training samples and found that our SFT+RL model performed better on the GSM8K dataset with varying compute budgets. Our main findings showed an overall higher accuracy while significantly reducing its token usage by over 40% compared to the SFT model, revealing how RL can recover the losses due to long-context training and altogether improving performance in mathematical reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。