arXiv:2508.13533cs.CLcs.LG2025-08

同一模型家族中大小模型不可盲目替换,信任度需独立评估。

Models in the Same Family are NOT Trust-Equivalent

  • 从特征依赖与置信度校准两维度评估模型信任等价性
  • 小大模型在输入特征依赖上差异显著,Jaccard系数低
  • 校准性能不随模型大小变化,大小模型均需单独验证

在同一模型家族中,当小型模型与大型模型性能相近时,常被直接替换使用。然而性能相似并不意味着信任等价。本文提出一个框架,从两个维度评估同家族模型间的信任等价性:一是归因一致性(是否基于相同输入特征做出预测),二是校准相似性(置信度与准确率的关系是否一致)。在Llama-2家族的自然语言推理与句子改写识别任务上,采用LIME和SHAP方法计算前K个关键特征的归因重合度,以杰卡德系数衡量一致性。结果表明,不同规模模型间归因一致性普遍较低,说明其决策依据不同。通过ECE、MCE、Brier Score及可靠性图分析校准性能,发现模型大小与校准质量无固定关系,校准表现差异显著。该结论在BERT和Vision Transformer两个纯编码器家族中亦得到验证。实验表明,模型替换需超越性能指标,信任等价性必须显式评估,不能仅凭性能相似就假设等价。

原文摘要 · Abstract (English)

Within a model family, a smaller variant is often deployed as a drop-in replacement for a larger one when their performance is similar. However, performance alone does not tell the full story. We propose a framework to evaluate trust-equivalence between a larger model and a smaller one in the same family along two dimensions. The first is attribution alignment: do both models base their predictions on the same input features? The second is calibration similarity: do both models share the same relationship between confidence and accuracy? We evaluate the Llama-2 family on two text classification tasks: Natural Language Inference and Paraphrase Identification. Attribution alignment is measured using two well-known methods: LIME and SHAP. Agreement between model pairs is quantified via the Jaccard coefficient over top-K attributed features. We observe that attribution alignment between models is generally low, indicating that smaller and larger models base their predictions on different input features. Calibration similarity is assessed using ECE, MCE, Brier Score, and Reliability Diagrams. Calibration profiles differ substantially across model sizes. There is no consistent relationship between model size and calibration quality. We have additionally verified these trends on two encoder-only families: BERT and Vision Transformer. The results are consistent with those reported here. Our experimental results show that replacing a larger model with a smaller one from the same family is a multidimensional decision that requires consideration beyond performance measures alone. Trust-equivalence must be assessed explicitly. It cannot be assumed from performance alone.

模型可解释性信任等价归因分析校准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。