用模型内部状态验证答案,速度更快更省资源。
HSRM: Hidden-State Reward Models for Test-Time Verification

- 直接读取生成模型的隐藏状态,不重读文本
- 在16个设置中15个表现超55M文本验证器
- 仅200万参数,适合高效推理场景
大语言模型常能生成看似合理的数学推理过程,但可靠识别正确解仍是关键挑战。现有测试时推理流程依赖文本验证器重新阅读每个生成解,导致验证成本高昂。已有研究表明,大模型在内部表示中已编码正确性信号,包括对自身答案错误可能性的认知。基于此,我们提出HSRM——一种轻量级隐藏状态奖励模型,通过直接读取生成器在推理步骤边界处的隐藏状态来验证候选解,而非重处理其文本。HSRM从冻结的生成器中提取隐藏状态,并使用小型Transformer编码器进行排序。该模型通过自生成轨迹与结果标签训练,无需人类编写过程监督或大型预训练验证器。在四个数学推理基准上,HSRM在16个生成器-数据集组合中的15个表现匹配或超越一个5500万参数的纯文本能量验证器,同时仅需约200万参数,通过复用生成过程中已计算的表示,为纯文本验证提供高效替代方案。
原文摘要 · Abstract (English)
Large language models can often generate plausible mathematical reasoning traces, but reliably identifying the correct solution among multiple candidates remains a key challenge. Existing test-time reasoning pipelines typically rely on text-based verifiers that re-read each generated solution, making verification an expensive component of inference. Prior work has shown, however, that LLMs often encode correctness-related signals in their internal representations, including awareness of when their own answers are likely to be wrong. Building on this observation, we introduce HSRM, a lightweight hidden-state reward model that verifies candidate solutions by directly reading the generator's internal representations rather than re-processing its text. HSRM extracts hidden states from a frozen generator at reasoning-step boundaries and uses a small Transformer encoder to rank candidates. It is trained from self-generated trajectories with outcome labels, requiring neither human-written process supervision nor a large pretrained verifier. Across four mathematical reasoning benchmarks, HSRM matches or outperforms a 55M-parameter text-only energy verifier in 15 of 16 generator--dataset settings while using only about 2M parameters, providing an efficient alternative to text-only verification by reusing representations already computed during generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。