用对比互信息自动标注推理步骤奖励，大幅降低标注成本。
Efficient Process Reward Modeling via Contrastive Mutual Information
- 基于模型内部概率计算步骤与答案的互信息增益
- 相比蒙特卡洛方法减少98%的生成token和84%时间
- 适合需要高效训练推理模型的研究者
近期研究致力于通过过程奖励模型（PRM）等验证思维链（CoT）轨迹中的中间推理步骤。然而，训练PRM通常需人工为每个推理步骤打分，成本高且耗时。现有自动化方法如蒙特卡洛（MC）估计也因重复大语言模型滚动生成而消耗大量计算资源。为此，本文提出对比点互信息（CPMI），一种新型自动奖励标注方法，利用模型内部概率推断步骤级监督，显著降低数据标注计算开销。CPMI衡量某一推理步骤相对于硬负例使步骤与正确答案间互信息增加的程度，该对比信号可作为步骤对最终解题贡献的代理奖励，结果可靠。实验表明，基于CPMI的标注相比MC估计，数据构建时间减少84%，生成token量减少98%，同时在过程级评估和数学推理基准上取得更高准确率。
原文摘要 · Abstract (English)
Recent research has devoted considerable effort to verifying the intermediate reasoning steps of chain-of-thought (CoT) trajectories using process reward models (PRMs) and other verifier models. However, training a PRM typically requires human annotators to assign reward scores to each reasoning step, which is both costly and time-consuming. Existing automated approaches, such as Monte Carlo (MC) estimation, also demand substantial computational resources due to repeated LLM rollouts. To overcome these limitations, we propose contrastive pointwise mutual information (CPMI), a novel automatic reward labeling method that leverages the model's internal probability to infer step-level supervision while significantly reducing the computational burden of annotating dataset. CPMI quantifies how much a reasoning step increases the mutual information between the step and the correct target answer relative to hard-negative alternatives. This contrastive signal serves as a proxy for the step's contribution to the final solution and yields a reliable reward. The experimental results show that CPMI-based labeling reduces dataset construction time by 84% and token generation by 98% compared to MC estimation, while achieving higher accuracy on process-level evaluations and mathematical reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。