VADE通过在线评估样本难度,动态选择高信息量样本,提升多模态强化学习效率。
VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RL
- 基于贝塔分布在线估算样本难度,用汤普森采样最大化信息增益
- 在多个基准上性能超越基线,样本效率提升且计算开销显著降低
- 可无缝接入现有方法,适合追求高效训练的多模态强化学习研究者
基于分组的策略优化方法(如GRPO和GSPO)已成为训练多模态模型的标准方案,利用分组回放与相对优势估计。然而,当组内所有响应获得相同奖励时,会引发严重的梯度消失问题,导致优势估计坍塌、训练信号减弱。现有缓解方法分为两类:基于过滤的方法先广泛建立回放,再事后剔除无信息组,带来巨大计算开销;基于采样的方法虽能预先选择有效样本,但依赖静态标准或历史数据知识,缺乏实时适应性。为此,我们提出VADE(Variance-Aware Dynamic Sampling via Online Sample-level Difficulty Estimation),一个基于在线样本级难度估计的方差感知动态采样框架。该框架包含三个核心组件:使用贝塔分布进行在线样本级难度估计,基于估计正确概率的汤普森采样器以最大化信息增益,以及双尺度先验衰减机制,确保在策略演化中保持稳健估计。这一设计使VADE能动态选取最具信息量的样本,从而增强训练信号且不增加额外回放成本。大量实验表明,VADE在多模态推理基准上持续优于强基线,在性能与样本效率上均表现更优,同时大幅降低计算开销。更重要的是,该框架可作为即插即用模块,无缝集成至现有分组式强化学习算法中。代码与模型见https://VADE-RL.github.io。
原文摘要 · Abstract (English)
Group-based policy optimization methods like GRPO and GSPO have become standard for training multimodal models, leveraging group-wise rollouts and relative advantage estimation. However, they suffer from a critical \emph{gradient vanishing} problem when all responses within a group receive identical rewards, causing advantage estimates to collapse and training signals to diminish. Existing attempts to mitigate this issue fall into two paradigms: filtering-based and sampling-based methods. Filtering-based methods first generate rollouts broadly and then retroactively filter out uninformative groups, leading to substantial computational overhead. Sampling-based methods proactively select effective samples before rollout but rely on static criteria or prior dataset knowledge, lacking real-time adaptability. To address these issues, we propose \textbf{VADE}, a \textbf{V}ariance-\textbf{A}ware \textbf{D}ynamic sampling framework via online sample-level difficulty \textbf{E}stimation. Our framework integrates three key components: online sample-level difficulty estimation using Beta distributions, a Thompson sampler that maximizes information gain through the estimated correctness probability, and a two-scale prior decay mechanism that maintains robust estimation under policy evolution. This three components design enables VADE to dynamically select the most informative samples, thereby amplifying training signals while eliminating extra rollout costs. Extensive experiments on multimodal reasoning benchmarks show that VADE consistently outperforms strong baselines in both performance and sample efficiency, while achieving a dramatic reduction in computational overhead. More importantly, our framework can serves as a plug-and-play component to be seamlessly integrated into existing group-based RL algorithms. Code and models are available at https://VADE-RL.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。