用模型内部状态探测推理可信度，实现高效测试时扩展。
ReProbe: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models
- 通过探测大模型内部状态，轻量级验证每步推理可靠性。
- 仅需不到1000万参数，性能媲美810倍大的奖励模型。
- 适合追求高效推理优化与模型可解释性的研究者。
大语言模型可通过生成长链多步推理解决复杂任务。测试时扩展（TTS）通过采样多个中间推理变体、验证其正确性并选择最优步骤继续，进一步提升性能。然而现有验证方法如过程奖励模型（PRM）计算开销大，且需大规模人工或模型标注。本文提出一种基于探测大模型内部状态的轻量级步骤级推理验证方法。训练一个仅含不足1000万参数的Transformer探针，利用冻结大模型的内部状态评估其推理步骤的可信度。标注可由更大模型（如DeepSeek-R1）提供，或由原模型自监督生成。在数学、规划和通用知识问答等多个领域，该探针性能匹配甚至超过规模高达810倍的PRM。结果表明，大模型内部状态蕴含推理置信度信息，可作为可靠步骤验证信号，为可扩展、泛化性强的TTS及更内省的大模型开辟新路径。
原文摘要 · Abstract (English)
LLMs can solve complex tasks by generating long, multi-step reasoning chains. Test-time scaling (TTS) can further improve performance by sampling multiple variants of intermediate reasoning steps, verifying their correctness, and selecting the best steps for continuation. However, existing verification approaches, such as Process Reward Models (PRMs), are computationally expensive and require large-scale human or model-generated annotations. We propose a lightweight alternative for step-level reasoning verification based on probing the internal states of LLMs. We train a transformer-based probe that uses the internal states of a frozen LLM to estimate the credibility of its reasoning steps during generation. Annotation can be provided either by a larger LLM (e.g., DeepSeek-R1) or in a self-supervised manner by the original model itself. The probes are lightweight, containing fewer than 10M parameters. Across multiple domains, including mathematics, planning, and general knowledge question answering, our probes match or exceed the performance of PRMs that are up to 810x larger. These results suggest that LLM internal states encode confidence in their reasoning processes and can serve as reliable signals for step verification, offering a promising path toward scalable, generalizable TTS and more introspective LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。