arXiv:2607.08456cs.CLcs.AI2026-07被引 2

大模型需区分答错与不该答的问题,本文发现二者是独立维度。

Two Axes of LLM Abstention: Answer Correctness and Question Answerability

  • 用隐藏状态探针检测问题可答性,比答案置信度更有效
  • 在错误前提问题上,传统方法准确率接近随机,探针达0.69-0.77 AUROC
  • 通过双轴校准策略,可分别控制答错率和不该答的比例

模型应拒绝两类情况:自身会答错的答案,以及本就不该回答的问题(如无解或基于错误前提)。现有方法仅依赖单一置信度阈值,无法区分二者。我们在三类共五款指令微调模型(2B至14B)上发现,这两者是独立维度。常规答案置信度能预测答案正确性,但对问题可答性几乎无感知;而隐藏状态线性探针则反之。该盲区不随模型规模增大而缩小,尤其在自然发生错误前提问题(CREPE)上表现最差——答案置信度、P(IK)、P(True)及直接询问前提是否错误均接近随机水平,而隐藏状态探针达到0.69–0.77 AUROC,表明模型内部已表征出问题却未报告。该问题可修复:直接要求检查前提反而引发57%的误挑战(混淆真伪前提),而结合探针路由后,挑战精度提升近三倍。我们构建双轴校准策略,分别验证可答性与正确性得分,实现更精准控制:在所有规模下均可调节不应答比例,且错误回答率被模型准确率上限约束,当覆盖率达0.75时,双阈值策略性能优于单阈值(0.31),14B模型中唯一实现有效认证。

原文摘要 · Abstract (English)

A model should refuse two different things: answers it would get wrong, and questions it should not answer at all, such as unanswerable ones or ones resting on a false premise. The usual recipe thresholds a single confidence score, which cannot tell these apart. Across five instruction-tuned models from three families (2B to 14B), we find they are separate axes. Ordinary answer-confidence tracks whether an answer is right but is nearly blind to whether the question is answerable; a linear probe on hidden states does the reverse. The blind spot does not shrink with scale. It is worst on naturally occurring false-premise questions (CREPE). There, answer-confidence, P(IK), P(True), and even asking the model outright whether a premise is false all stay near chance, while a hidden-state probe reaches 0.69 to 0.77 AUROC: the model represents a problem it will not report. This turns out to be fixable. Instructing a model to check premises backfires, because it then disputes sound and false premises alike (57% false challenges), unable to tell them apart; routing the same instruction with the probe roughly triples challenge precision. We turn the two axes into a calibrated policy that answers only when an answerability score and a correctness score each clear a separately certifies behave differently: the unanswerable-answer rate is controllable at every scale, while the wrong-answer rate is capped by model accuracy, so the guarantee tightens as threshold policy certifies both budgets at 0.75 coverage of correct answers, against 0.31 for a single threshold; at 14B it is the only policy that certifies at all.

大模型拒答机制可答性双轴校准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。