给黑箱大模型的每一步推理打分，找错更准。
Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
- 基于推理过程的一致性，逐步评估可信度。
- 在数学和问答任务中，低可信度步骤与错误强相关。
- 适合想改进模型推理、调试故障的研究者。
大型语言模型在需要客观答案的推理任务中表现强劲，通过生成分步解题过程实现，但诊断多步推理链中何处出错仍具挑战。置信度估计可提供诊断信号，但现有方法受限于最终答案或需访问内部模型。本文提出步骤置信度归属（SCA）框架，仅基于生成的推理轨迹为闭源大模型分配步骤级置信度。SCA采用信息瓶颈原理：与正确解一致结构对齐的步骤获得高置信度，偏离则被标记为可能出错。提出两种互补方法：(1) NIBS，一种无参数的信息瓶颈方法，不依赖图结构衡量一致性；(2) GIBS，基于图的信息瓶颈模型，通过可微掩码学习子图以捕捉逻辑变异性。在数学推理和多跳问答任务上的大量实验表明，SCA能可靠识别低置信度步骤，其与推理错误高度相关。此外，利用步骤级置信度引导自我修正，相较仅基于答案反馈，修正成功率最高提升13.5%。
原文摘要 · Abstract (English)
Large Language Models have achieved strong performance on reasoning tasks with objective answers by generating step-by-step solutions, but diagnosing where a multi-step reasoning trace might fail remains difficult. Confidence estimation offers a diagnostic signal, yet existing methods are restricted to final answers or require internal model access. In this paper, we introduce Stepwise Confidence Attribution (SCA), a framework for closed-source LLMs that assigns step-level confidence based only on generated reasoning traces. SCA applies the Information Bottleneck principle: steps aligning with consensus structures across correct solutions receive high confidence, while deviations are flagged as potentially erroneous. We propose two complementary methods: (1) NIBS, a non-parametric IB approach measuring consistency without graph structures, and (2) GIBS, a graph-based IB model that learns subgraphs through a differentiable mask to capture logical variability. Extensive experiments on mathematical reasoning and multi-hop question answering show that SCA reliably identifies low-confidence steps strongly correlated with reasoning errors. Moreover, using step-level confidence to guide self-correction improves the correction success rate by up to 13.5\% over answer-level feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。