arXiv:2409.16292cs.CVcs.AI2024-09被引 2

用热力图解释人类对图像相似性的判断,揭示关键视觉区域。

Explaining Human Comparisons using Alignment-Importance Heatmaps

  • 通过深度模型提取特征图的重要度得分,量化其对人机表征对齐的贡献。
  • 仅用高分特征图构建表示,可显著提升对新样本人类判断的预测精度。
  • 热力图直观展示比较时的关键区域,适合研究视觉认知与模型可解释性。

我们提出一种用于人类图像比较任务的计算可解释性方法,基于深度视觉模型生成的对齐重要性得分(AIS)热力图。AIS反映特征图在使深度神经网络(DNN)表征几何与人类表征几何对齐中的独特贡献。首先验证了AIS的有效性:在训练集上筛选出高AIS得分的特征图构建表示后,对未见样本的人类相似性判断预测性能显著提升。随后,计算每张图像对应的热力图,直观显示与高AIS特征图相关的图像区域。这些区域对应于图像比较中的关键信息,与眼动预测模型生成的显著性图存在对应关系。但部分情况下,比较相关维度并非最显著的视觉区域。结果表明,对齐重要性不仅提升了从DNN嵌入中预测人类相似性判断的能力,还为图像空间中的相关信息提供了可解释洞察。

原文摘要 · Abstract (English)

We present a computational explainability approach for human comparison tasks, using Alignment Importance Score (AIS) heatmaps derived from deep-vision models. The AIS reflects a feature-map's unique contribution to the alignment between Deep Neural Network's (DNN) representational geometry and that of humans. We first validate the AIS by showing that prediction of out-of-sample human similarity judgments is improved when constructing representations using only higher-scoring AIS feature maps identified from a training set. We then compute image-specific heatmaps that visually indicate the areas that correspond to feature-maps with higher AIS scores. These maps provide an intuitive explanation of which image areas are more important when it is compared to other images in a cohort. We observe a correspondence between these heatmaps and saliency maps produced by a gaze-prediction model. However, in some cases, meaningful differences emerge, as the dimensions relevant for comparison are not necessarily the most visually salient. To conclude, Alignment Importance improves prediction of human similarity judgments from DNN embeddings, and provides interpretable insights into the relevant information in image space.

可解释性图像对比特征重要性深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。