提出Token Insight方法,识别医学影像模型决策中的关键特征片段。
Identifying Critical Tokens for Accurate Predictions in Transformer-based Medical Imaging Models
- 通过逐个移除输入片段(token)来评估其对预测的影响。
- 在结肠息肉识别任务中验证了模型决策的可解释性提升。
- 无需额外模块,适用于任意视觉变压器模型,适合临床可信AI研发。
随着自监督学习(SSL)的发展,基于Transformer的计算机视觉模型在性能上已超越卷积神经网络(CNN),有望在未来主导人工智能医学影像领域。然而,与CNN类似,揭示基于Transformer模型的决策机制仍是挑战。本文提出一种名为Token Insight的新方法,旨在解析基于Transformer的医学影像模型的决策过程,通过利用Transformer原生的片段丢弃机制,无需额外模块即可识别影响预测的关键输入片段(token)。该方法量化每个token对预测的贡献,实现对模型决策更细致的理解。实验结果表明,在使用监督和自监督预训练的视觉变压器进行结肠息肉识别任务时,Token Insight显著提升了模型的透明度与可解释性,有助于增强临床信任并推动其在实际医疗场景中的应用。
原文摘要 · Abstract (English)
With the advancements in self-supervised learning (SSL), transformer-based computer vision models have recently demonstrated superior results compared to convolutional neural networks (CNNs) and are poised to dominate the field of artificial intelligence (AI)-based medical imaging in the upcoming years. Nevertheless, similar to CNNs, unveiling the decision-making process of transformer-based models remains a challenge. In this work, we take a step towards demystifying the decision-making process of transformer-based medical imaging models and propose Token Insight, a novel method that identifies the critical tokens that contribute to the prediction made by the model. Our method relies on the principled approach of token discarding native to transformer-based models, requires no additional module, and can be applied to any transformer model. Using the proposed approach, we quantify the importance of each token based on its contribution to the prediction and enable a more nuanced understanding of the model's decisions. Our experimental results which are showcased on the problem of colonic polyp identification using both supervised and self-supervised pretrained vision transformers indicate that Token Insight contributes to a more transparent and interpretable transformer-based medical imaging model, fostering trust and facilitating broader adoption in clinical settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。