让小模型学会在不确定时放弃回答，提升判断可靠性。
Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models
- 通过添加‘我不知道’选项观察回答稳定性来检测不确定性。
- 在4个数据集上实现最高10.81%的风险改善，小模型表现更优。
- 无需额外参数，适合资源受限的部署场景，尤其适合低性能模型。
大型语言模型常在不确定时仍给出自信但错误的答案，这一问题在计算资源受限的小语言模型（SLMs）中尤为突出。本文提出「Second Guess」——一种轻量级、无需参数的提示技术，用于多选题问答中的主动放弃（abstention）。核心发现是：真正知道答案的模型会稳定选择同一答案，而不确定的模型在加入‘我不知道’选项后会出现回答不稳定现象。在四个开源小模型（2B-8B参数）和四个基准测试上评估，该方法实现最高的复合风险改善达10.81%。值得注意的是，即使在微调模型上仍保持8%的风险改善，而基于熵的方法在此类模型上性能下降。该方法对低性能模型提升最显著。所有代码与复现结果见https://github.com/Mystic-Slice/second-guess。
原文摘要 · Abstract (English)
Large language models often generate confident but incorrect answers rather than abstaining when uncertain. This problem is particularly acute for small language models (SLMs), where computational constraints and autonomous operation amplify the need for reliable uncertainty detection. We propose _Second Guess_, a lightweight, parameter-free prompting technique for abstention in multiple-choice question answering (MCQA) that is well-suited for SLMs. Our key empirical insight is that models which truly know an answer will select it consistently, while uncertain models exhibit unstable behavior when an ``I don't know'' option is added. Evaluated on four open models (2B-8B parameters) and four benchmarks, Second Guess achieves the highest composite risk improvement of 10.81\%. Notably, it maintains an 8\% composite risk improvement on fine-tuned models where entropy-based methods degrade, and improves most for lower-performing models. All code and results required to reproduce this work is available in https://github.com/Mystic-Slice/second-guess
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。