arXiv:2603.12151cs.LGcs.AI2026-03被引 3

提出LLM强化学习采样计算的最优分配方案,提升训练效率。

IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

  • 通过三资源优化框架,确定采样并行数随算力增长规律。
  • 采样并行数随算力增加而上升,最终趋于饱和,适用于难易问题。
  • 方法可直接指导实际训练,适合追求高效微调的研究者。

尽管缩放定律能指导大语言模型预训练的算力分配,但其后训练阶段强化学习(RL)的算力配置仍缺乏明确准则。本文研究基于在线策略强化学习的大语言模型采样算力最优分配,将缩放问题建模为三个资源的算力约束优化:每问题的并行采样数量、每批次的问题数及更新步数。研究发现,最优并行采样数随算力预算增长而递增,随后趋于饱和。该趋势在简单与复杂问题中均成立,但机制不同:简单问题因解集锐化,复杂问题则因覆盖扩展。进一步表明,增加并行采样可缓解问题间干扰,而每批次问题数主要影响训练稳定性,可在较宽范围内灵活选择。结果在多种基础模型与数据分布上验证有效,将强化学习缩放规律转化为可操作的资源配置指南,为大语言模型高效后训练提供实用支持。

原文摘要 · Abstract (English)

While scaling laws guide compute allocation for LLM pre-training, analogous prescriptions for reinforcement learning (RL) post-training of large language models (LLMs) remain poorly understood. We study the compute-optimal allocation of sampling compute for on-policy RL methods in LLMs, framing scaling as a compute-constrained optimization over three resources: parallel rollouts per problem, number of problems per batch, and number of update steps. We find that the compute-optimal number of parallel rollouts per problem increases predictably with compute budget and then saturates. This trend holds across both easy and hard problems, though driven by different mechanisms: solution sharpening on easy problems and coverage expansion on hard problems. We further show that increasing the number of parallel rollouts mitigates interference across problems, while the number of problems per batch primarily affects training stability and can be chosen within a broad range. Validated across base models and data distributions, our results recast RL scaling laws as prescriptive allocation rules and provide practical guidance for compute-efficient LLM RL post-training.

强化学习大模型算力优化采样

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。