用主动学习减少标注量，让推理模型更高效地训练过程奖励模型。
Efficient Process Reward Model Training via Active Learning
- 基于不确定性选择样本，仅标注最需改进的步骤数据
- 标注量减少50%仍达相当或更好性能，数学推理数据筛选保留60%
- 适合追求标注效率与高精度奖励模型的研究者
过程奖励模型（PRM）为大语言模型提供逐步监督，但大规模数据标注对人工和模型均具挑战。为此，我们提出主动学习方法ActPRM，主动选取最不确定样本进行标注，显著降低标注成本。训练中，利用PRM在前向传播后估计不确定性，仅保留高不确定数据，再由高性能但耗时的推理模型标注。随后计算损失并更新PRM权重。在池基主动学习设置下对比传统微调，ActPRM减少50%标注量，性能相当甚至更优。进一步，使用ActPRM过滤超100万条数学推理轨迹，保留60%数据；基于此数据集训练的新模型在ProcessBench（75.0%）和PRMBench（65.5%）上达到同规模模型最优表现。
原文摘要 · Abstract (English)
Process Reward Models (PRMs) provide step-level supervision to large language models (LLMs), but scaling up training data annotation remains challenging for both humans and LLMs. To address this limitation, we propose an active learning approach, ActPRM, which proactively selects the most uncertain samples for training, substantially reducing labeling costs. During training, we use the PRM to estimate uncertainty after the forward pass, retaining only highly uncertain data. A capable yet costly reasoning model then labels this data. Then we compute the loss with respect to the labels and update the PRM's weights. We compare ActPRM vs. vanilla fine-tuning, on a pool-based active learning setting, demonstrating that ActPRM reduces 50% annotation, but achieving the comparable or even better performance. Beyond annotation efficiency, we further advance the actively trained PRM by filtering over 1M+ math reasoning trajectories with ActPRM, retaining 60% of the data. A subsequent training on this selected dataset yields a new state-of-the-art (SOTA) PRM on ProcessBench (75.0%) and PRMBench (65.5%) compared with same sized models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。