用现成大模型做推理评分，无需训练就能提升小模型数学推理能力。
Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning
- 用大模型对小模型生成的固定长度片段评分，避免错误传播。
- 在多个数学数据集上超越多数投票，接近甚至超过需训练的强化学习方法。
- 无需奖励模型训练，适合资源有限但追求高精度推理的场景。
使用更强模型在生成过程中评分候选延续路径是提升响应质量的有效策略，但当小模型已陷入错误推理路径时，该方法失效。基于奖励模型的引导搜索（PRM）虽可避免此问题，但需步骤级标注进行训练。本文提出一种无需训练的分块引导生成框架，利用现成大语言模型作为过程评分器。每步中，小模型生成k个固定长度候选片段，大模型通过似然值评分而不生成文本，选择最优片段推进下一步，防止错误累积。提出两种策略：最大似然选择（LGS）和对比选择（CGS），后者通过减去小模型概率，突出大模型偏好与小模型的差异。实验表明，变量长度评分存在系统性长度偏差，而固定长度片段可规避此问题。在GSM8K、MATH、Minerva Math、AMC23、AIME24等数据集上，以Qwen2.5-1.5B为小模型、Qwen2.5-32B为评分器，CGS相比多数投票最高提升28个百分点；在相同引导预算下，表现媲美或优于训练过的Qwen2.5-Math-PRM-72B。当以Qwen2.5-7B为小模型、Qwen2.5-72B为评分器时，CGS在MATH上达81.8%，在Minerva Math上达63.6%（k=16），比多数投票高出4–6个百分点。此外，该方法生成的推理路径显著短于PRM引导搜索。
原文摘要 · Abstract (English)
Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths. PRM guided search avoids this by scoring candidate continuations during generation, but requires a reward model trained with step-level labels. We propose Chunk-Level Guided Generation, a training-free alternative that uses an off-the-shelf large language model as a process scorer. At each step, a small model samples k fixed-length candidate chunks, while the larger model scores the candidates using likelihoods without generating any text. The selected chunk is committed before the next step, steering generation before errors can propagate. We instantiate this framework with two selection rules: Likelihood-Guided Selection (LGS), which selects the chunk with the highest length-normalized large-model log-probability, and Contrastive-Guided Selection (CGS), which subtracts the small model's log-probability to favor chunks where the large model's preference diverges from the small model's. We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias that persists even after length normalization, and that fixed-length chunks avoid this confound. On GSM8K, MATH, Minerva Math, AMC23, and AIME24 with Qwen2.5-1.5B guided by Qwen2.5-32B and Llama-3.2-1B guided by Llama-3.1-70B, CGS outperforms majority voting by up to 28 pp and, under matched guidance budgets, matches or outperforms Qwen2.5-Math-PRM-72B guided search on most benchmarks without reward-model training. With Qwen2.5-7B guided by Qwen2.5-72B, CGS reaches 81.8% on MATH and 63.6% on Minerva Math at k=16, surpassing majority voting by 4--6 pp. Finally, Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。