解析视觉GNN的决策逻辑,揭示其与人类认知的差异
Explaining Vision GNNs: A Semantic and Visual Analysis of Graph-based Image Classification
- 通过分析不同层图结构的语义一致性,揭示模型推理机制
- 发现深层网络的决策过程与人类感知不一致,存在偏差
- 结合热力图可视化,提升GNN图像分类的可解释性
图神经网络(GNN)在图像分类等视觉任务中成为卷积方法的有效替代,采用图像块作为节点构建图结构,边基于块间相似性或分类相关性建立。尽管图结构本身具备天然可解释性,但基于GNN的视觉模型的可解释性仍研究不足。本文分析了视觉GNN分类器各层图结构的语义一致性,重点关注其对物体结构和有意义关系的保持程度。通过量化层间连接反映语义相似性和空间连贯性的程度,进行系统评估。同时,对比标准与对抗样本下的解释结果,检验模型鲁棒性。此外,利用热力图可视化技术展示信息在层间的流动路径,揭示模型可解释性。实验表明,这些模型的决策过程可被有效解释,但深层网络的推理机制与人类感知并不一致。
原文摘要 · Abstract (English)
Graph Neural Networks (GNNs) have emerged as an efficient alternative to convolutional approaches for vision tasks such as image classification, leveraging patch-based representations instead of raw pixels. These methods construct graphs where image patches serve as nodes, and edges are established based on patch similarity or classification relevance. Despite their efficiency, the explainability of GNN-based vision models remains underexplored, even though graphs are naturally interpretable. In this work, we analyze the semantic consistency of the graphs formed at different layers of GNN-based image classifiers, focusing on how well they preserve object structures and meaningful relationships. A comprehensive analysis is presented by quantifying the extent to which inter-layer graph connections reflect semantic similarity and spatial coherence. Explanations from standard and adversarial settings are also compared to assess whether they reflect the classifiers' robustness. Additionally, we visualize the flow of information across layers through heatmap-based visualization techniques, thereby highlighting the models' explainability. Our findings demonstrate that the decision-making processes of these models can be effectively explained, while also revealing that their reasoning does not necessarily align with human perception, especially in deeper layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。