arXiv:2604.18094cs.CV2026-04

让视觉Transformer的解释更精准,关键在关注决策相关的注意力传播。

Decision-Aware Attention Propagation for Vision Transformer Explainability

论文配图:Decision-Aware Attention Propagation for Vision Transformer Explainability
图 1 · 摘自论文原文
  • 将决策相关先验注入注意力传播,融合梯度定位与层级注意力机制。
  • 生成的注意力图更具类别区分性、更紧凑且更忠实于最终预测结果。
  • 适合需要高精度解释的ViT模型研究者,尤其关注可解释性提升。

视觉变压器(ViTs)在计算机视觉中已成为主流架构,但其预测过程难以解释,因信息通过多层和注意力头间的复杂交互传播。现有基于注意力的解释方法依赖原始注意力权重,未能明确反映最终决策,导致解释的类别区分度有限。相比之下,基于梯度的定位方法能更好突出类别特异性证据,但未充分利用变压器的层次化注意力传播机制。为此,本文提出决策感知注意力传播(DAP),通过梯度定位估计标记重要性,并将其整合至逐层注意力回传中,同时捕捉注意力的结构传播路径与对最终预测最相关的证据。实验表明,DAP在不同规模的ViT变体上均显著优于现有基线,在定量指标和定性可视化上表现更优,证明决策感知传播是提升ViT可解释性的有效方向。

原文摘要 · Abstract (English)

Vision Transformers (ViTs) have become a dominant architecture in computer vision, yet their prediction process remains difficult to interpret because information is propagated through complex interactions across layers and attention heads. Existing attention based explanation methods provide an intuitive way to trace information flow. However, they rely mainly on raw attention weights, which do not explicitly reflect the final decision and often lead to explanations with limited class discriminability. In contrast, gradient based localization methods are more effective at highlighting class specific evidence, but they do not fully exploit the hierarchical attention propagation mechanism of transformers. To address this limitation, we propose Decision-Aware Attention Propagation (DAP), an attribution method that injects decision-relevant priors into transformer attention propagation. By estimating token importance through gradient based localization and integrating it into layer wise attention rollout, the method captures both the structural flow of attention and the evidence most relevant to the final prediction. Consequently, DAP produces attribution maps that are more class sensitive, compact, and faithful than those generated by conventional attention based methods. Extensive experiments across Vision Transformer variants of different model scales show that DAP consistently outperforms existing baselines in both quantitative metrics and qualitative visualizations, indicating that decision aware propagation is an effective direction for improving ViT interpretability.

可解释性视觉Transformer注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。