量化模型对相似输入的解释稳定性,发现隐藏偏见与推理不一致。
Empirical Characterization of Rationale Stability Under Controlled Perturbations for Explainable Pattern Recognition

- 用余弦相似度衡量同标签样本的SHAP值一致性,检测解释波动。
- BERT在SST-2上表现更稳定,但存在对特定词汇的依赖偏差。
- 适合关注模型可解释性可信度的研究者和工程师。
可靠的模式识别系统应在相似输入下表现出一致行为,其解释也应保持稳定。然而,当前可解释AI评估多聚焦单个样本,未明确量化同一类别或微小变化输入间的归因模式一致性。本文提出一种新度量方法,通过预训练BERT在SST-2情感分析数据集上计算SHAP特征重要性,并在RoBERTa、DistilBERT及IMDB数据集上进行鲁棒性测试。该度量以同标签样本间SHAP值的余弦相似度为核心,用于检测模型是否对特定特征产生偏倚依赖或在相似预测中推理不一致。实验表明,该方法能有效识别模型行为与预期目标的偏离,优于传统保真度指标。该框架为模型行为提供更稳健的验证手段,对构建可信AI系统至关重要。代码已公开于https://github.com/anmspro/ESS-XAI-Stability。
原文摘要 · Abstract (English)
Reliable pattern recognition systems should exhibit consistent behavior across similar inputs, and their explanations should remain stable. However, most Explainable AI evaluations remain instance centric and do not explicitly quantify whether attribution patterns are consistent across samples that share the same class or represent small variations of the same input. In this work, we propose a novel metric aimed at assessing the consistency of model explanations, ensuring that models consistently reflect the intended objectives and consistency under label-preserving perturbations. We implement this metric using a pre-trained BERT model on the SST-2 sentiment analysis dataset, with additional robustness tests on RoBERTa, DistilBERT, and IMDB, applying SHAP to compute feature importance for various test samples. The proposed metric quantifies the cosine similarity of SHAP values for inputs with the same label, aiming to detect inconsistent behaviors, such as biased reliance on certain features or failure to maintain consistent reasoning for similar predictions. Through a series of experiments, we evaluate the ability of this metric to identify misaligned predictions and inconsistencies in model explanations. These experiments are compared against standard fidelity metrics to assess whether the new metric can effectively identify when a model's behavior deviates from its intended objectives. The proposed framework provides a deeper understanding of model behavior by enabling more robust verification of rationale stability, which is critical for building trustworthy AI systems. By quantifying whether models rely on consistent attribution patterns for similar inputs, the proposed approach supports more robust evaluation of model behavior in practical pattern recognition pipelines. Our code is publicly available at https://github.com/anmspro/ESS-XAI-Stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。