小模型靠自评信心判断是否该求助大模型,省钱省时还能保持高准确率。
Do Small Language Models Know When They're Wrong? Confidence-Based Cascade Scoring for Educational Assessment

- 让小模型自评信心值,用作决定是否交给大模型处理的信号。
- 最佳方案成本降76%、延迟降61%,准确率接近大模型(kappa 0.802 vs 0.819)。
- 信心区分度是关键:信心分布平庸的小模型无法提升效果。
大规模自动化评分需在准确性、成本与延迟间权衡。在“级联”系统中,小语言模型处理简单任务,复杂任务则移交大模型,但难点在于何时升级。本文探索将模型自述信心(预测时附带数值置信度)作为路由信号。基于2,100条学生-人工智能数学对话的专家评分数据,评估了GPT-5.4、Claude 4.5+和Gemini 3.1三组模型对构建的级联系统。结果表明:(1) 小模型间的信心区分能力差异显著,最优者达到AUROC 0.857,最差者信心分布近乎无差别;(2) 模型信心与人工评分难度正相关,当标注员分歧大或耗时长时,模型信心更低;(3) 最优级联系统以76%的成本降低和61%的延迟降低,逼近大模型精度(kappa 0.802 vs 0.819)。信心区分能力是瓶颈:仅两个具备有效信心方差的小模型实现了无统计显著误差的级联,而第三个信心近似恒定的模型无论如何调阈值也无法缩小准确率差距。具备强区分力的小模型可实现成本与准确率的灵活权衡,否则无法受益。
原文摘要 · Abstract (English)
Automated scoring of student work at scale requires balancing accuracy against cost and latency. In "cascade" systems, small language models (LMs) handle easier scoring tasks while escalating harder ones to larger LMs -- but the challenge is determining which cases to escalate. We explore verbalized confidence -- asking the LM to state a numerical confidence alongside its prediction -- as a routing signal. Using 2,100 expert-scored decisions from student-AI math conversations, we evaluate cascade systems built from GPT-5.4, Claude 4.5+, and Gemini 3.1 model pairs. We find that: (1) confidence discrimination varies widely across small LMs, with the best achieving AUROC 0.857 and the worst producing a near-degenerate confidence distribution; (2) confidence tracks human scoring difficulty, with lower LM confidence where annotators disagreed and took longer to score; (3) the best cascade approached large-LM accuracy (kappa 0.802 vs. 0.819) at 76% lower cost and 61% lower latency. Confidence discrimination is the bottleneck: the two small LMs with meaningful confidence variance yielded cascades with no statistically detectable kappa loss, while the third -- whose confidence was near-degenerate -- could not close the accuracy gap regardless of threshold. Small LMs with strong discrimination let practitioners trade cost for accuracy along the frontier; those without it do not.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。