通过自适应难度调度，让模型高效学数学推理。
Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
- 根据模型表现动态调整题目难度，保持训练挑战性
- 在数学竞赛数据集上训练时间减少最多2倍
- 无需改架构或奖励函数，适合想提速的LLM研究者
强化微调（RFT）在提升大语言模型数学推理能力方面潜力巨大，但通常样本和计算效率低下，需大量训练。本文提出AdaRFT（自适应课程强化微调），通过基于模型近期奖励信号动态调整训练题目的难度，确保模型始终在具有挑战性但可解的任务上训练。该自适应采样策略维持最优难度区间，避免对过易或过难问题浪费计算资源。AdaRFT仅需对标准RFT算法（如PPO）进行轻量级扩展，不修改奖励函数或模型结构。在竞赛级数学数据集上的实验表明，AdaRFT显著提升收敛效率与推理性能；在具备题目难度标注的情况下，训练时间最多缩短2倍，适用于不同数据分布与模型规模，提供更可扩展、高效的RFT框架。
原文摘要 · Abstract (English)
Reinforcement finetuning (RFT) has shown great potential for enhancing the mathematical reasoning capabilities of large language models (LLMs), but it is often sample- and compute-inefficient, requiring extensive training. In this work, we introduce AdaRFT (Adaptive Curriculum Reinforcement Finetuning), a method that significantly improves the efficiency of RFT through adaptive curriculum learning. AdaRFT dynamically adjusts the difficulty of training problems based on the model's recent reward signals, ensuring that the model consistently trains on tasks that are challenging but solvable. This adaptive sampling strategy accelerates learning by maintaining an optimal difficulty range, avoiding wasted computation on problems that are too easy or too hard. AdaRFT requires only a lightweight extension to standard RFT algorithms like Proximal Policy Optimization (PPO), without modifying the reward function or model architecture. Experiments on competition-level math datasets demonstrate that AdaRFT improves convergence efficiency and reasoning performance. Given problem-level difficulty annotations, AdaRFT reduces RFT training time by up to 2 times across data distributions and model scales, offering a more scalable and effective RFT framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。