用大模型内部隐藏状态做推理轨迹评分，高效选出最优解。
TrajSelector: Harnessing Latent Representations for Efficient and Effective Best-of-N in Large Reasoning Model
- 用轻量验证器基于模型隐状态评估每步推理质量
- 32选1下比多数投票高4.61%，比现有方法高4.31%~12.21%
- 无需标注数据，适合追求高效推理的场景
大型语言模型在复杂推理任务中表现突出，主要得益于测试时扩展（TTS）范式，尤其是外部TTS中的最佳-32选择（Best-of-N）机制，通过从多个独立生成的推理路径中筛选最优解实现性能提升。然而该方法存在两大局限：一是部署过程奖励模型带来的高计算开销，二是未能充分利用大模型内在的隐状态表示。本文提出TrajSelector框架，利用采样模型的隐藏状态进行过程级评分。该框架采用仅0.6B参数的轻量级验证器，对每一步推理路径进行质量评估，并聚合得分以识别最优推理路径。整个流程采用全数据驱动、端到端训练方式，无需依赖大规模步骤级标注。在五个基准测试中，实验结果表明，TrajSelector在最佳-32设置下，准确率相比多数投票提升4.61%，优于现有过程奖励模型4.31%至12.21%，同时保持更低的推理成本。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown remarkable progress in complex reasoning tasks, largely enabled by test-time scaling (TTS) paradigms that allocate additional compute during inference. Among these, external TTS (particularly the Best-of-N selection paradigm) yields scalable performance improvements by selecting from multiple independently generated reasoning trajectories. However, this approach faces key limitations: (i) the high computational overhead of deploying process reward models, (ii) the underutilization of the LLM's intrinsic latent representations. We introduce TrajSelector, an efficient and effective Best-of-N framework that exploit the hidden states in the sampler LLM for process-level scoring. A lightweight verifier (with only 0.6B parameters) evaluates the quality of step-wise trajectory, and then aggregates these scores to identify the optimal reasoning trajectory. Our framework employs a fully data-driven, end-to-end training recipe that eliminates reliance on massive step-level annotations. Experiential results across five benchmarks demonstrate that TrajSelector delivers consistent performance gains. In Best-of-32 settings, it surpasses majority voting by 4.61% accuracy and outperforms existing process reward models by 4.31% to 12.21%, all while maintaining lower inference costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。