用简单数据训练大模型,逐步攻克难题,效率远超传统方法
Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learning

- 用少量标注数据启动,通过分步策略处理难样本
- 仅用10%易标注数据,数学与科学任务表现超越顶尖基线
- 适合资源有限但需持续提升推理能力的模型优化场景
以往基于大模型的强化学习研究多依赖高成本标注或无监督投票/熵奖励机制,但受限于标注开销及模型坍塌、奖励欺骗等问题,性能不佳。本文受认知学习理论启发,提出全新方法 EasyRL:通过模拟人类知识获取曲线,将简单标注数据中的可靠知识迁移与渐进式分治策略结合,逐步解决更难的未标注数据。具体而言,先用少样本标注数据初始化模型;再对困难样本采用一致性选择(低不确定性)与反思式解析(中等不确定性)的伪标签策略;最后通过难度递进的自训练与强化学习迭代增强推理能力。EasyRL 构建了一个统一的自演化框架,实现高效后训练。在数学与科学基准测试中,仅使用10%的易标注数据,便持续优于现有最佳方法。
原文摘要 · Abstract (English)
Previous LLMs-based RL studies typically follow either supervised learning with high annotation costs, or unsupervised paradigms using voting or entropy-based rewards. However, their performance remains far from satisfactory due to the substantial annotation cost and issues such as model collapse or reward hacking. To address these issues, we introduce a new perspective inspired by cognitive learning theory and propose a novel approach called EasyRL. The core of EasyRL is to simulate the human cognitive acquisition curve by integrating reliable knowledge transfer from easy labeled data with a progressive divide-and-conquer strategy that tackles increasingly difficult unlabeled data. Specifically, we initialize a warm-up model using supervised RL with few-shot labeled data. This is followed by a divide-and-conquer pseudo-labeling strategy on difficult unlabeled data, combining consistency-based selection for low-uncertainty cases and reflection-based resolution for medium-uncertainty cases. Finally, difficulty-progressive self-training with iterative pseudo-labeling and RL further strengthens the model's reasoning capability. EasyRL provides a unified self-evolving framework that facilitates data-efficient post-training of LLMs. Experimental results on mathematical and scientific benchmarks demonstrate that EasyRL, using only 10% of easy labeled data, consistently outperforms state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。