提出预测训练规律的方法，用更少算力实现大模型推理微调
Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models
- 基于模型规模和初始表现建立训练动态预测模型
- 发现训练分三个阶段，过早停止可省70%以上算力
- 适用于不同模型，为强化学习微调提供高效实践指南
使用强化学习方法（如分组相对策略优化，GRPO）微调大型语言模型进行推理任务计算开销巨大。为此，我们提出一个预测框架，建模训练动态并优化资源使用。在Llama和Qwen模型（3B、8B）上的实验表明，基于模型规模、初始性能和训练进度的实证缩放定律能预测奖励轨迹，并识别出三个一致的训练阶段：缓慢启动、快速提升和平台期。我们发现，超过一定轮次后继续训练收益极低，提示提前终止可显著减少计算量而不影响性能。该方法在不同模型类型间具有泛化性，为基于GRPO的微调提供了实用高效的指导。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) for reasoning tasks using reinforcement learning methods like Group Relative Policy Optimization (GRPO) is computationally expensive. To address this, we propose a predictive framework that models training dynamics and helps optimize resource usage. Through experiments on Llama and Qwen models (3B 8B), we derive an empirical scaling law based on model size, initial performance, and training progress. This law predicts reward trajectories and identifies three consistent training phases: slow start, rapid improvement, and plateau. We find that training beyond certain number of an epoch offers little gain, suggesting earlier stopping can significantly reduce compute without sacrificing performance. Our approach generalizes across model types, providing a practical guide for efficient GRPO-based fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。