标注预算有限时，选模型难的样本能大幅提效。
Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgets
- 只用最难10%样本训练，性能提升最高达47%。
- 简单样本训练后性能仅提升3-15%，因缺乏学习信号。
- 适合资源受限下优化强化学习微调的实践者。
语言模型微调的数据收集成本高昂，实际预算限制了可获取数据量。本文通过对比不同难度样本（简单、中等、困难、随机）在多个模型和推理任务上的表现，研究样本难度对GRPO训练效果的影响。结果显示，仅使用基础模型最常失败的前10%最难样本进行训练，性能提升最高可达47%；而简单样本仅带来3%-15%的微弱改进。这是因为GRPO依赖结果方差生成学习信号：难样本在整个训练过程中保持成功与失败的混合状态，而简单样本很快收敛为一致成功，导致学习信号消失。此外，仅在难样本上训练的模型在AIME2025基准上表现出显著的分布外泛化能力，实现了有意义的性能提升。结论明确：预算受限时，应优先收集并标注模型难以处理的样本，这些样本贡献了几乎全部的训练价值。
原文摘要 · Abstract (English)
Collecting high-quality training examples for language model fine-tuning is expensive, with practical budgets limiting the amount of data that can be procured. We investigate whether example difficulty affects GRPO training effectiveness by comparing selection strategies (easy, medium, hard, random) across multiple models and reasoning tasks. Training on the hardest 10\% of examples (those where the base model fails most often) yields dramatic performance gains up to 47\%, while easy examples produce minimal improvements of 3-15\%. This occurs because GRPO requires outcome variance to generate learning signals; hard examples maintain mixed success/failure outcomes throughout training while easy examples quickly converge to consistent success, eliminating learning opportunities. Moreover, models trained on hard examples show superior out-of-distribution generalization, with only hard-trained models achieving meaningful gains on the AIME2025 benchmark. Our findings provide clear guidance: when budget-constrained, prioritize collecting and annotating examples where your base model struggles, as these drive nearly all learning value in GRPO fine-tuning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。