单一提示准确率会掩盖模型可靠性问题,需多变体评估。
What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models
- 设计不同提示变体,系统测试模型在多个任务中的表现
- 72%-88%准确率下降源于评估器缺陷而非模型本身
- 模型大小与提示鲁棒性无稳定关联,需报告评估细节
单提示准确率是主流语言模型评估方式,但可能遗漏关键可靠性问题。本文评估15个开源模型,重点分析10个指令型模型在五类分类与推理基准上,每项数据集搭配五种提示变体的性能。全面测量准确率、标记概率校准、口头信心校准、口头解析率及提示扰动范围。结果发现:首先,评估设计显著影响结论;将期望校准误差(ECE)从原始定义改为标签集归一化后,平均绝对校准差异达0.149;更惊人的是,在ARC-Challenge上使用思维链提示配合首字符评估器,主模型准确率下降72%-88%,两种修复方法恢复了93.8%和102.7%性能,表明问题出在评估器而非模型。其次,信心信号极脆弱:在MMLU-Pro上,所有主模型口头信心远高于其准确率和标记概率信心,且单个模型在特定提示下解析率崩溃。第三,模型参数量与提示鲁棒性无可靠相关性,10个指令模型在各基准上的相关系数介于-0.244至0.474之间。综合表明,小模型的可靠性结论不仅取决于模型本身,还高度依赖评估流程。建议在做可靠性声明时,明确报告校准定义、评估逻辑、口头可解析性及提示鲁棒性。
原文摘要 · Abstract (English)
Single-prompt accuracy is the dominant way to benchmark language models, but it can miss reliability failures that matter. We evaluate a 15-model open-weight corpus, with the main reliability analyses focused on 10 instruct models across five classification and reasoning benchmarks under five prompt variants each, measuring accuracy, token-probability calibration, verbal-confidence calibration, verbal parse rate, and prompt-perturbation spread for every (model x dataset x variant) cell. We find three broad results. First, evaluation design can materially change the conclusion. Switching Expected Calibration Error (ECE) token from a raw to a label-set-normalised definition changes per-cell calibration by a mean absolute 0.149. More strikingly, pairing a chain-of-thought prompt with a first-character evaluator on ARC-Challenge reduces apparent accuracy by 72-88% across all five primary models; two independent repair procedures recover 93.8% and 102.7% of the lost performance, indicating an evaluator-side rather than model-side failure. Second, confidence signals are fragile. On MMLU-Pro, every primary model verbally reports confidence substantially above both its accuracy and its token-probability confidence on the same rows, and verbal parse rate can collapse for a single model on a single prompt variant. Third, prompt robustness does not track parameter count reliably. Across 10 instruct models, the correlation between model size and prompt-perturbation spread ranges from -0.244 to 0.474 across benchmarks. Taken together, these results show that reliability conclusions for small language models depend not only on the model being evaluated, but also on the evaluation pipeline used to measure it. We argue that calibration definitions, evaluator logic, verbal parseability, and prompt robustness should be reported explicitly when making reliability claims.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。