用自然语言批评指导大模型推理，提升准确率且无需训练。
Process Supervision via Verbal Critique Improves Reasoning in Large Language Models
- 通过生成-批评-优化循环，用强模型的自然语言反馈指导弱模型。
- 在GPQA钻石题上达94.9%准确率，超越现有方法且不需梯度更新。
- 适合想提升推理能力又无法微调模型的研究者和工程师。
大语言模型推理的推理时扩展主要关注三个维度：链式深度、样本广度和学习到的步骤评分器（PRMs）。本文提出第四维——外部自然语言监督的粒度，即言语过程监督（VPS），一种无需训练的框架，利用更强模型的结构化自然语言批评，引导迭代生成-批评-修正循环，最多进行R轮。在GPQA Diamond、AIME 2025和LiveCodeBench V6（涵盖封闭与开放模型）上，VPS取得三项关键结果：第一，在GPQA Diamond上，GPT-5.4（高）| GPT-5.4（低）在R=4时达到94.9%，超过94.1%的现有最佳水平，且无梯度更新；第二，在AIME 2025上，实现强弱模型救援，得分从11.7%-26.7%提升至63.3%-90.0%（最高+63.3分）；第三，在相同算力下，相比Reflexion提升+8.5至+12.1分，相比Self-Consistency@5提升+5.0个百分点（GPQA）和+8.3个百分点（LiveCodeBench），证明批评粒度是核心驱动力。性能随监督者与执行者能力差距增大而上升（皮尔逊相关系数r=0.90），当错误无法用语言表达时（如代码合成）性能下降，推动混合语言-可执行方法的发展。这些结果确立了批评粒度作为推理时扩展的新维度。
原文摘要 · Abstract (English)
Inference-time scaling for LLM reasoning has focused on three axes: chain depth, sample breadth, and learned step-scorers (PRMs). We introduce a fourth axis, granularity of external verbal supervision, via Verbal Process Supervision (VPS), a training-free framework that uses structured natural-language critique from a stronger supervisor to guide an iterative generate-critique-refine loop up to a round budget R. Across GPQA Diamond, AIME 2025, and LiveCodeBench V6 (covering both closed and open models), VPS yields three key results. First, on GPQA Diamond, GPT-5.4 (High) | GPT-5.4 (Low) reaches 94.9% at R=4, surpassing the 94.1% state of the art without gradient updates. Second, on AIME 2025, VPS enables strong weak-actor rescue, boosting scores from 11.7-26.7% to 63.3-90.0% (up to +63.3 points). Third, at matched compute, VPS outperforms Reflexion by +8.5 to +12.1 points and Self-Consistency@5 by +5.0 pp (GPQA) and +8.3 pp (LiveCodeBench), isolating critique granularity as the key driver. Performance scales with the supervisor-actor capability gap (Pearson r=0.90) and degrades when errors are not linguistically expressible (e.g., code synthesis), motivating hybrid verbal-executable methods. These results establish critique granularity as a new axis of inference-time scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。