arXiv:2412.15828cs.LG2024-12AAAI被引 18

提出可量化多模态交互的解释方法,让医疗AI决策更透明

Measuring Cross-Modal Interactions in Multimodal Models

  • 基于Shapley交互指数,精准分离各模态及其交互贡献
  • 支持多模态、无需标签数据,能对单个病例生成详细解释
  • 适用于医疗场景,提升复杂模型的可解释性与可信度

将AI引入医疗可显著提升患者护理质量和系统效率,但现有AI系统的可解释性(XAI)不足,尤其在使用复杂多模态模型时更为突出。多数现有XAI方法仅针对单模态模型,无法捕捉多源数据间的交叉影响。现有交叉模态交互量化方法局限于双模态,依赖标注数据且与模型性能相关,这在医疗领域难以满足个性化解释需求。本文提出InterSHAP,一种新型交叉模态交互评分方法,利用Shapley交互指数精确分解各模态及其交互贡献,无需近似。通过与开源SHAP工具包集成,提升可复现性和易用性。实验表明,InterSHAP能准确衡量交叉模态交互存在性,支持多模态输入,并为个体样本提供局部详细解释。进一步在多模态医疗数据集上验证了其在个性化解释中的适用性。

原文摘要 · Abstract (English)

Integrating AI in healthcare can greatly improve patient care and system efficiency. However, the lack of explainability in AI systems (XAI) hinders their clinical adoption, especially in multimodal settings that use increasingly complex model architectures. Most existing XAI methods focus on unimodal models, which fail to capture cross-modal interactions crucial for understanding the combined impact of multiple data sources. Existing methods for quantifying cross-modal interactions are limited to two modalities, rely on labelled data, and depend on model performance. This is problematic in healthcare, where XAI must handle multiple data sources and provide individualised explanations. This paper introduces InterSHAP, a cross-modal interaction score that addresses the limitations of existing approaches. InterSHAP uses the Shapley interaction index to precisely separate and quantify the contributions of the individual modalities and their interactions without approximations. By integrating an open-source implementation with the SHAP package, we enhance reproducibility and ease of use. We show that InterSHAP accurately measures the presence of cross-modal interactions, can handle multiple modalities, and provides detailed explanations at a local level for individual samples. Furthermore, we apply InterSHAP to multimodal medical datasets and demonstrate its applicability for individualised explanations.

可解释AI多模态医疗AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。