动态调整训练数据顺序，让大模型推理能力提升5.2%。
Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning
- 按语义分组并动态调整样本分布，避免盲目采样。
- 优先选策略边界样本，提升训练信息量，平均准确率提升5.2%。
- 适配多种强化学习目标，适合所有后训练阶段的模型优化。
大语言模型通过强化学习后训练获得强大推理能力，但现有方法多采用均匀采样，忽视数据语义结构与策略能力变化。为此，本文提出自适应数据调度（ADS），一种双层调度框架，以语义聚类和策略边界样本选择替代均匀采样。在聚类层面，根据语义模式组织样本，并动态调整跨聚类分布，巩固当前训练进展；在样本层面，持续采样策略边界样本，提供高信息量的相对优势信号。在三个大模型与七个推理基准上的实验表明，相较于组相对策略优化（GRPO），ADS平均准确率提升5.2%。且对不同目标设计的强化学习方法均有增益，展现出作为通用数据调度策略的潜力。代码已开源：https://github.com/Richard-zrx/ADS。
原文摘要 · Abstract (English)
Large Language Models (LLMs) achieve remarkable reasoning capabilities through reinforcement learning (RL) post-training. However, existing RL post-training commonly relies on uniform data sampling, which ignores the semantic structure of the training data and the changing capability of the training policy. To address these limitations, we propose Adaptive Data Scheduling (ADS), a dual-level data scheduling framework for pacing RL post-training that replaces uniform sampling with an adaptive distribution over semantic clusters and policy-boundary sample selection. At the cluster level, ADS organizes samples according to semantic patterns and maintains an adaptive inter-cluster distribution to solidify current training progress. At the sample level, ADS performs intra-cluster scheduling to continuously sample policy-boundary samples, which provides informative relative advantages. Experimental results across three LLMs and seven reasoning benchmarks demonstrate that ADS improves average accuracy by 5.2% over Group Relative Policy Optimization (GRPO). Notably, ADS consistently improves RL methods with different objective designs, highlighting its potential as a general data scheduling strategy for LLM RL post-training. The source code is available at: https://github.com/Richard-zrx/ADS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。