用图结构共享推理难度信息,让大模型更高效地学习。
Efficient RLVR Scheduling via Graph-Structured Online Difficulty Estimation

- 构建语义相似样本的图结构,共享探索反馈
- 在线更新难度估计,避免冷启动和过时信息
- 无需额外探测,适合各类强化学习推理场景
基于可验证奖励的强化学习(RLVR)能提升大语言模型的推理能力,但依赖昂贵的滚动探索。对不同难度样本分配相同探索预算效率低下:简单样本可能冗余探索,困难但可学样本则探索不足。现有自适应调度器通过课程学习或非均匀分配解决此问题,但在线难度估计仍具挑战——专用探测增加生成开销,历史依赖型估计存在冷启动与反馈滞留问题,且忽略样本间关联。为此,我们提出即插即用的图结构在线难度估计器,通过相关样本间共享探索反馈,持续更新难度估计,缓解冷启动与滞留问题,无需专用探测。具体地,基于语义与推理相似性构建难度感知样本图,引入潜在难度状态,并用Potts先验促使邻近样本共享状态;再使用状态级贝塔-二项分布模型聚合各状态的滚动结果;最后通过在线均值场变分算法持续更新潜状态分配与状态级难度。该框架可集成至样本选择与滚动分配调度器中,实现无探测的难度自适应探索。多模型、多调度器、多基准测试结果表明,本框架性能更优。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models but relies on costly rollout exploration. Assigning the same exploration budget to samples with different difficulty levels is inefficient: easy samples may receive redundant rollouts, whereas difficult but learnable samples may receive too little exploration. Existing adaptive schedulers address this mismatch through curriculum-based sample selection or non-uniform rollout allocation based on estimated sample difficulty. However, obtaining reliable online difficulty estimates remains challenging: dedicated probing adds substantial generation overhead, whereas history-based estimators face a cold start with no initial observations and stale feedback, and typically ignore relations among samples. To address these limitations, we propose a plug-and-play graph-based online difficulty estimator that shares rollout feedback across related samples and continuously updates their difficulty estimates, mitigating cold start and staleness without dedicated probing. Specifically, we first construct a difficulty-aware sample graph based on semantic and reasoning similarities. Based on this graph, we introduce latent difficulty states and use a Potts prior to encourage neighboring samples to share the same state. We then employ a state-level Beta-Binomial model to aggregate the rollout outcomes associated with each state. Finally, we use an online mean-field variational algorithm to continuously update the latent-state assignments and state-level difficulty as new feedback arrives. Our framework can be integrated into sample-selection and rollout-allocation schedulers, enabling difficulty-adaptive exploration without dedicated probing. Experiments across multiple base models, RL schedulers, and benchmarks demonstrate that our framework achieves better performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。