用信息论自动标注推理步骤质量，提升大模型链式思考的可靠性。
Process Supervision for Chain-of-Thought Reasoning via Monte Carlo Net Information Gain
- 基于信息论计算每步对答案的影响，生成自动生成步骤评分。
- 在数学、编程等任务中，最佳K选一评估下准确率显著提升。
- 计算复杂度降为O(N)，比之前方法更高效，适合大规模应用。
多步推理虽增强大语言模型能力，但易导致错误传播。过程奖励模型（PRMs）通过逐步评分实现细粒度监督，提升可靠性。现有训练方法依赖昂贵的人工标注或高计算成本的自动标记。本文提出一种基于信息论的自动步骤标签生成方法，估算每一步对正确答案概率的影响，从而提供步骤质量信号。该方法将计算复杂度降至$/mathcal{O}(N)$，优于先前的$/mathcal{O}(N \log N)$。实验证明，这些标签可在多样化的推理基准（包括数学、Python编程、SQL及科学问答）中有效支持best-of-$K$选择，提升链式思考的准确性。本工作实现了可扩展、高效的模型推理监督，尤其适用于错误传播敏感的任务。
原文摘要 · Abstract (English)
Multi-step reasoning improves the capabilities of large language models (LLMs) but increases the risk of errors propagating through intermediate steps. Process reward models (PRMs) mitigate this by scoring each step individually, enabling fine-grained supervision and improved reliability. Existing methods for training PRMs rely on costly human annotations or computationally intensive automatic labeling. We propose a novel approach to automatically generate step-level labels using Information Theory. Our method estimates how each reasoning step affects the likelihood of the correct answer, providing a signal of step quality. Importantly, it reduces computational complexity to $\mathcal{O}(N)$, improving over the previous $\mathcal{O}(N \log N)$ methods. We demonstrate that these labels enable effective chain-of-thought selection in best-of-$K$ evaluation settings across diverse reasoning benchmarks, including mathematics, Python programming, SQL, and scientific question answering. This work enables scalable and efficient supervision of LLM reasoning, particularly for tasks where error propagation is critical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。