追踪强化学习数据源头,发现多数数据来自少数原始来源。
RLVR Datasets and Where to Find Them: Tracing Data Lineage for Better Training Data

- 通过溯源搜索框架ATLAS,定位145万条数据的20个原始来源。
- 99.7%的数据可追溯至原始来源,多数数据为已有数据的变体。
- 提出新数据集DAPO++和质量评分Q,提升训练有效性与可预测性。
强化学习从可验证奖励(RLVR)数据集的泛滥导致数据溯源模糊,引发溯源崩溃问题。为弥合碎片化的RLVR数据生态,我们提出原子源溯源框架ATLAS,能将145万条实例中超过99.7%回溯至20个原始来源。分析显示,多数RLVR数据集是少数上游数据的变体,鲜有真正新增内容,且存在数据污染风险。据此,我们构建新数据集DAPO++,并提出基于源级反事实归因(SCA)的指导原则,通过比较各原子源对应的强化学习检查点与共享基础模型,衡量样本边际效用。基于此归因信号,设计复合数据质量评分Q,其与下游RLVR性能强相关。在Qwen3系列模型上的实验表明,DAPO++在保留测试集上持续提升性能,而Q可可靠预测训练效果。代码与数据已开源。
原文摘要 · Abstract (English)
The proliferation of Reinforcement Learning from Verifiable Rewards (RLVR) datasets has exacerbated provenance collapse due to unclear lineage among existing datasets. To bridge this fragmented RLVR data landscape, we propose Atomic-source Tracing via Lineage-Aware Search (ATLAS), a systematic framework for tracing RLVR datasets back to their atomic sources, attributing over 99.7% of 1.45M instances to 20 atomic sources. Our analysis reveals that most RLVR datasets are variants of a small set of shared upstream sources, with few introducing genuinely new data, and many facing data contamination risks. These findings naturally motivate us to curate a new RLVR dataset, DAPO++, and to benchmark existing datasets from a lineage-aware perspective. To this end, we propose Source-level Counterfactual Attribution (SCA) as a guiding principle to curate a decontaminated training dataset with concentrated learning signals. Essentially, SCA measures a sample's marginal utility by comparing per-atomic-source RL checkpoints against a shared base model. Building upon these attribution signals, we further design a composite dataset quality score Q that strongly correlates with downstream RLVR performance. Experiments on Qwen3 series models verify that DAPO++ consistently improves performance on held-out benchmarks, while Q reliably predicts downstream RLVR training effectiveness. Our code and data is available at https://github.com/Celine-hxy/ATLAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。