arXiv:2509.22115cs.LGcs.AI2025-09被引 1

通过动态筛选关键样本和标记,用更少数据实现更强的强化学习优化。

Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization

  • 分样本与标记两级筛选,优先保留高价值信息。
  • 减少30%以上样本和标记,性能超越现有方法。
  • 适合追求高效训练的强化学习研究者使用。

Critic-free 方法如 GRPO 通过多次采样估算优势值以降低内存需求,但因大量无信息样本和标记导致收敛缓慢。为此,本文提出动态双层降采样框架 D³S,通过两级筛选提升策略优化效率。在样本层面,选择能最大化优势方差(Var(A))的采样序列,理论上与策略梯度范数上界正相关,从而获得更强梯度信号;在标记层面,优先处理优势值绝对值与策略熵乘积高的标记,聚焦于既不确定又影响大的位置。为防止对高信号数据过拟合,D³S 采用受课程学习启发的动态降采样策略:初期激进降采样加速早期学习,后期逐步放松以提升泛化能力。在 Qwen2.5 与 Llama3.1 上的广泛实验表明,将 D³S 集成至先进强化学习算法,在多种推理基准上均实现领先性能与泛化能力,同时显著减少所需样本与标记数量。代码已附于补充材料,将公开发布。

原文摘要 · Abstract (English)

Critic-free methods like GRPO reduce memory demands by estimating advantages from multiple rollouts but tend to converge slowly, as critical learning signals are diluted by an abundance of uninformative samples and tokens. To tackle this challenge, we propose the \textbf{Dynamic Dual-Level Down-Sampling (D$^3$S)} framework that prioritizes the most informative samples and tokens across groups to improve the efficient of policy optimization. D$^3$S operates along two levels: (1) the sample-level, which selects a subset of rollouts to maximize advantage variance ($\text{Var}(A)$). We theoretically proven that this selection is positively correlated with the upper bound of the policy gradient norms, yielding higher policy gradients. (2) the token-level, which prioritizes tokens with a high product of advantage magnitude and policy entropy ($|A_{i,t}|\times H_{i,t}$), focusing updates on tokens where the policy is both uncertain and impactful. Moreover, to prevent overfitting to high-signal data, D$^3$S employs a dynamic down-sampling schedule inspired by curriculum learning. This schedule starts with aggressive down-sampling to accelerate early learning and gradually relaxes to promote robust generalization. Extensive experiments on Qwen2.5 and Llama3.1 demonstrate that integrating D$^3$S into advanced RL algorithms achieves state-of-the-art performance and generalization while requiring \textit{fewer} samples and tokens across diverse reasoning benchmarks. Our code is added in the supplementary materials and will be made publicly available.

强化学习高效训练降采样

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。