arXiv:2609.06107cs.LGcs.CL2026-09

对比13种数据策略对强化学习训练效果的影响,发现均匀采样仍最可靠。

DataFlex-RL: An Evaluation Platform for RLVR Data Policies

论文配图:DataFlex-RL: An Evaluation Platform for RLVR Data Policies
图 1 · 摘自论文原文
  • 在统一训练流程下测试多种数据选择与加权方法
  • 均匀采样比初始模型提升7.76个百分点准确率
  • 无一种策略能稳定超越均匀采样,且评估结果受基准集影响大

针对带可验证奖励的强化学习(RLVR)中的数据策略,我们提出DataFlex-RL评估平台,用于在相同GRPO框架下比较不同策略。主实验在12个匹配种子上使用Qwen2.5-7B-Base和12个数学、逻辑、科学基准测试了13种配置。均匀GRPO相比未训练检查点,使领域平衡平均准确率提升7.76个百分点。八种回溯采样或重加权方法均未在95%置信区间内显著优于均匀采样;三种自适应混合策略也未优于固定等权重混合。基于Llama-3.1-8B-Base的修正12种子扩展将新方法置于同一评分尺度,但未发现性能一致更优的方法。通过数学密集型六基准集(含五数学题与GPQA-Diamond,不含逻辑题)重评分九次Qwen2.5-7B-Instruct运行,与12基准领域平衡汇总排名呈负相关(相关系数-0.33),而保留全部12基准的汇总结果高度一致。在受控条件下,数据策略显著影响训练过程,但无法稳定带来超越均匀采样的可复现提升。

原文摘要 · Abstract (English)

Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which domains contribute to subsequent training batches. We introduce DataFlex-RL, an evaluation platform for comparing these choices under a common GRPO recipe. Our primary experiment evaluates 13 configurations across 12 matched seeds using Qwen2.5-7B-Base and 12 mathematics, logic, and science benchmarks. Uniform GRPO improves the domain-balanced average accuracy by 7.76 percentage points over the untrained checkpoint. None of the eight rollout-selection or reweighting methods achieves a paired 95% confidence interval that excludes zero relative to uniform sampling, and none of the three adaptive mixtures outperforms a fixed equal mixture at the same level of precision. A corrected 12-seed extension on Llama-3.1-8B-Base places the additional methods on the same score scale as the original controls, but does not reveal a consistent winner in terms of observed mean performance. We also quantify evaluation sensitivity by rescoring nine Qwen2.5-7B-Instruct runs using a math-heavy six-benchmark summary, consisting of five mathematics benchmarks and GPQA-Diamond but no logic benchmark, and comparing it with the domain-balanced 12-benchmark summary. The resulting rankings are negatively correlated, with a correlation coefficient of -0.33, whereas summaries that retain all 12 benchmarks largely agree. Across the controlled settings studied here, changing the data policy measurably changes the training process but does not produce a reproducible improvement over uniform training.

强化学习数据策略评估平台可验证奖励

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。