通过主动调整生成分布，让大模型强化学习更高效
DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning
- 根据响应分布特性，主动筛选简洁确定的生成路径
- 实验显示提速最高达1.77倍，性能无损
- 适合追求推理效率的LLM强化学习研究者
强化学习已成为提升模型能力的关键，但其采样效率受长尾响应长度分布制约。现有方法多通过提示层尾部调度缓解问题，本文则聚焦于效率低下的根源——分布本身。我们细粒度刻画分布特性，发现提示内存在长尾现象，且常由无效冗余内容构成。为此，提出一种分布感知的主动轨迹塑形新范式，通过分布感知采样机制从冗余探索空间中选择路径，并设计自适应冗余分配策略，在保证塑形效果的同时提升系统效率。实验表明，该方法在不牺牲模型性能的前提下，相比最先进系统实现最高1.77倍的加速。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has become pivotal for improving model capabilities yet suffers from rollout efficiency bottlenecks due to the long-tail response length distribution. While existing works mitigate the impact of long tails via prompt-level tail scheduling, we focus on the root source of inefficiency: the distribution itself. Specifically, we characterize the long-tail distribution at a finer granularity, identifying intra-prompt long tails, and revealing that they frequently consist of ineffective verbosity. To address this, we propose a novel paradigm of active distribution shaping to shape the rollout distribution towards conciseness and certainty, thereby fundamentally resolving tail-induced overheads. We achieve this through a distribution-aware trajectory sampling mechanism, which selects trajectories from a redundant exploration space for each prompt, and an adaptive redundancy allocation scheme to maximize both shaping effectiveness and system efficiency. Experiments demonstrate significant acceleration over state-of-the-art systems by up to 1.77x without compromising model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。