用对抗生成错误训练奖励模型，提升大模型推理鲁棒性。
Adversarial Training for Process Reward Models
- 生成器制造推理错误，奖励模型学习识别，双方迭代优化。
- 在数学推理任务上比最强基线提升3.4个百分点，分布外任务提升5.3个百分点。
- 无需人工标注步骤级标签，适合需要强泛化能力的场景。
过程奖励模型（PRMs）通过提供步骤级监督来增强大语言模型的推理能力。然而，其广泛应用受限于昂贵的手动步骤级标注以及静态训练数据对新错误的泛化能力差。我们提出对抗训练的PRMs（APRM），其中生成器（G）学习生成可欺骗奖励模型（R）的推理错误，而R则同步学习检测这些错误。这种交互使R不断面对更难的负样本，从而提升其对新错误的鲁棒性和泛化能力，且无需手动步骤级标注。在多个数学推理基准上，APRM平均比最强基线提升3.4个百分点；在分布外任务上提升5.3个百分点。
原文摘要 · Abstract (English)
Process Reward Models (PRMs) enhance reasoning ability of LLMs by providing step-level supervision. However, their widespread adoption is limited due to expensive manual step-level annotation and poor generalization of static training data to novel errors. We introduce Adversarially Trained PRMs (\texttt{APRM}), where a Generator ($G$) learns to produce reasoning errors to deceive a PRM ($R$), while $R$ concurrently learns to detect them. This interaction yields progressively harder negatives for $R$, improving its robustness and generalization to novel errors without requiring manual step-level labels. Averaged across diverse mathematical reasoning benchmarks, \texttt{APRM} improves solver accuracy by $+3.4$ percentage points (pp) over the strongest PRM baseline. \texttt{APRM} achieves gains of $+5.3$ pp on out-of-distribution tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。