将注意力机制融入解释框架,提升大模型可解释性。
Integrating attention into explanation frameworks for language and vision transformers
- 用注意力权重重构特征重要性计算,实现局部解释。
- 结合概念激活向量,全局衡量模型对语义的敏感度。
- 在自然语言与视觉任务中均验证有效,适合研究可解释性者。
注意力机制是Transformer架构的核心,其内部信号具有可解释性,激发了基于注意力的模型解释研究热潮。尽管注意力权重不直接决定输出,但能反映标记间的影响力模式,可补充现有可解释性技术。本文提出两种新方法,将注意力信息整合至不同面向模型行为的可解释人工智能(XAI)框架中。第一种方法通过注意力权重重新定义成对标记交互的特征函数,将广义的博弈论解法Shapley值调整为注意力驱动的局部归因;第二种方法将注意力权重引入基于概念激活向量的逐标记方向导数,用于衡量全局概念敏感性。在标准基准上的实证评估及与主流解释方法的对比表明,注意力权重可被有意义地融入所研究的XAI框架,凸显其在丰富Transformer可解释性方面的价值。
原文摘要 · Abstract (English)
The attention mechanism lies at the core of the transformer architecture, providing an interpretable model-internal signal that has motivated a growing interest in attention-based model explanations. Although attention weights do not directly determine model outputs, they reflect patterns of token influence that can inform and complement established explainability techniques. This work studies the potential of utilising the information encoded in attention weights to provide meaningful model explanations by integrating them into explainable AI (XAI) frameworks that target fundamentally different aspects of model behaviour. To this end, we develop two novel explanation methods applicable to both natural language processing and computer vision tasks. The first integrates attention weights into the Shapley value decomposition by redefining the characteristic function in terms of pairwise token interactions via attention weights, thus adapting this widely used game-theoretic solution concept to provide attention-driven attributions for local explanations. The second incorporates attention weights into token-level directional derivatives defined through concept activation vectors to measure concept sensitivity for global explanations. Our empirical evaluations on standard benchmarks and in a comparison study with widely used explanation methods show that attention weights can be meaningfully incorporated into the studied XAI frameworks, highlighting their value in enriching transformer explainability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。