根据模型能力动态分配计算资源，提升大模型强化学习效率
CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMs
- 用能力导向价值函数评估任务训练价值，指导资源分配
- 在多个基准上实现一致的泛化性能提升，超越均匀预算方法
- 适合追求高效大模型后训练的科研与工程人员
基于可验证奖励的强化学习（RLVR）已成为提升大模型推理能力的关键方法。然而，标准框架如组相对策略优化（GRPO）通常采用统一的回滚预算，导致资源利用效率低下。现有自适应方法多依赖实例级指标（如任务通过率），无法捕捉模型动态学习状态。为此，我们提出CoBA-RL，一种基于模型能力演化自适应分配回滚预算的强化学习算法。具体而言，CoBA-RL利用能力导向价值函数将任务映射至潜在训练增益，并采用堆结构贪心策略，高效地将计算资源分配给高训练价值样本。大量实验表明，该方法能有效协调探索与利用，在多个挑战性基准上实现一致的泛化性能提升。结果表明，量化样本训练价值并优化预算分配是提升大模型后训练效率的关键。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key approach for enhancing LLM reasoning. However, standard frameworks like Group Relative Policy Optimization (GRPO) typically employ a uniform rollout budget, leading to resource inefficiency. Moreover, existing adaptive methods often rely on instance-level metrics, such as task pass rates, failing to capture the model's dynamic learning state. To address these limitations, we propose CoBA-RL, a reinforcement learning algorithm designed to adaptively allocate rollout budgets based on the model's evolving capability. Specifically, CoBA-RL utilizes a Capability-Oriented Value function to map tasks to their potential training gains and employs a heap-based greedy strategy to efficiently self-calibrate the distribution of computational resources to samples with high training value. Extensive experiments demonstrate that our approach effectively orchestrates the trade-off between exploration and exploitation, delivering consistent generalization improvements across multiple challenging benchmarks. These findings underscore that quantifying sample training value and optimizing budget allocation are pivotal for advancing LLM post-training efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。