提出可预测隐藏状态选择是否优于投票的判据,提升大模型决策准确率。
A decodability criterion predicts when hidden-state selection beats majority voting in large language models

- 用线性门控机制从隐藏状态中选出最可能正确的答案
- 在中等难度题上准确率提升19点,难题上提升16.8点
- 新提出的可解度指标能提前判断该方法是否有效,适合部署前评估
大语言模型生成多个答案后如何融合决策是测试时信息融合问题,通常采用多数投票。但在难题上,样本答案易产生相关错误,导致投票失效,增加采样反而更差。通过读取模型隐藏状态中的正确性信号进行候选选择是一种替代方案,但其效果因模型和任务而异,缺乏可信的判断标准。本文提出CASE(Correctness-Axis SElection)动态选择组合器,通过训练线性门控机制对答案标记的隐藏状态打分并选最高分项。核心贡献是引入‘可解度’——一种无泄漏的衡量指标,反映门控能否将正确答案排在错误答案之上,可预测隐藏状态选择是否优于投票。传统探测器看似准确实因问题身份泄露,该现象在按问题组评估时消失。在未见数据上,可解度与选择相比投票的准确率提升呈0.75的皮尔逊相关性,决策阈值约在AUC=0.60。在通用与医学大模型中,案例在中等难度问题上提升最高达19分,在难题上提升16.8分。可解度取决于模型需调用的对齐知识,不依赖规模,且在未见过的科学领域仍保持3.8分的预测迁移能力。因此,它提供了可预先测量的实用判据,用于决定在特定任务中应采用学习式选择还是多数投票。
原文摘要 · Abstract (English)
Combining the answers a large language model (LLM) samples for a question into one decision is a test-time information fusion problem, usually solved by majority voting. Voting is unreliable on difficult questions, where the sampled answers share correlated errors, so the wrong answer can win and drawing more samples makes the decision worse. Selecting a candidate by reading a correctness signal from the model's hidden states is a promising alternative, but its accuracy varies across models and tasks, and no measure indicates when it can be trusted. In this paper, we propose CASE (Correctness-Axis SElection), a dynamic selection combiner that trains a linear gate on the answer-token hidden state and selects the highest-scoring candidate. Its main contribution is decodability, a leakage-free measure of how well the gate ranks a question's correct candidates above its incorrect ones, which predicts whether hidden-state selection will outperform voting. A conventional probe appears accurate only because of question-identity leakage, which vanishes under question-grouped evaluation. On held-out data, decodability predicts the accuracy gain of selection over voting with a Pearson correlation r=0.75 and a decision threshold near AUC=0.60. Across general and medical LLMs, CASE improves over voting by up to 19 points on medium-difficulty questions and 16.8 points on hard questions. Decodability depends on the aligned knowledge a model must recall, not on its scale, and its prediction transfers to an unseen scientific domain within 3.8 points. It thus provides a practical criterion, measurable in advance for a given model and task, for choosing between learned selection and majority voting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。