高效提升大模型推理能力,仅用10%标签即可保持性能。
DE-Venus: A Data-Efficient RLVR Framework for Large Language Models

- 将监督信号作为数据准备与训练的动态过程统一管理。
- 在公开基准和业务场景中,仅需10%标签或13%相关数据保持效果。
- 适合需要降本增效的大模型训练团队,尤其关注标注成本。
基于可验证奖励的强化学习(RLVR)能提升大语言模型的推理能力,但其实际扩展受限于高昂的在线策略采样成本及大规模可靠目标获取难度。现有方法分别处理样本选择、不完整监督或噪声标签,常将监督逻辑与分布式训练耦合,难以控制比较和复用。本文提出DE-Venus,一个数据高效的统一框架,将监督视为数据准备与策略优化中的演进状态。其包含三个模块:主动数据选择分配训练与标注预算;弱监督构建从无标签样本中提取学习信号;训练时监督精炼过滤或修正不可靠监督。DE-Venus支持七种代表性方法及数据选择流程,通过离线数据转换或在线目标、奖励、批次、优势的调整实现方法特异性决策,同时保留Verl的分布式执行契约。在多个公开基准与三个业务场景中,不同配置均以仅10%标签或最低13%相关数据维持甚至提升模型质量;部分业务配置使收敛步数减少63%–75%。该框架在不牺牲可扩展强化学习执行的前提下,显著降低标注与训练成本。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning, but its practical scaling is constrained by expensive on-policy rollouts and the cost of obtaining reliable targets at scale. Existing methods address sample selection, incomplete supervision, or noisy labels separately, often entangling supervision logic with distributed training and hindering controlled comparison and reuse. We present DE-Venus, a unified framework for data-efficient RLVR that treats supervision as evolving state across data preparation and policy optimization. It organizes this lifecycle into three modules: Active Data Selection allocates training and annotation budgets; Weak Supervision Construction derives learning signals from unlabeled examples; and Training-Time Supervision Refinement filters or corrects unreliable supervision. DE-Venus supports seven representative methods and a data-selection pipeline by expressing method-specific decisions as offline dataset transitions or online transformations of targets, rewards, batches, and advantages while preserving verl's distributed execution contracts. Across public benchmarks and three business scenarios, separate configurations preserve or improve model quality with only 10% of labels or as little as 13% of relevant data; selected business configurations also reduce observed convergence steps by 63%--75%. DE-Venus thus reduces annotation and training costs without sacrificing scalable RL execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。