用语法树整合医学术语,让多模态模型解释更准确可信。
Explaining BiomedCLIP with Weighted Banzhaf Interactions Supported by Tree-Gram Parsing

- 用依赖句法树将分散的医学词组合并成语义单元作为解释基础。
- 在ROCOv2数据集上,解释结果语义连贯性显著提升,噪声减少。
- 适合需要可解释性的医疗AI场景,如放射科辅助诊断。
视觉语言模型(VLMs)在放射学等医学任务中展现出强大能力,但其决策过程缺乏可信且可解释的说明,限制了临床部署。现有方法如FIxLIP常因现代分词器的细粒度特性而失效——例如将“saddle embolus”拆分为无意义子词,导致跨模态归因混乱。本文提出ParseFIxLIP,将树形短语解析(Tree-Gram Parsing)引入FIxLIP的Banzhaf交互博弈中。该方法基于spaCy依赖句法树,通过smart_depth策略聚合相关文本词元,形成语义完整的概念单元,有效缓解概念碎片化问题。定量结果显示,尽管基线方法在长描述下遭遇高维挑战,本方法仍保持统计稳健与语义简洁性。在BiomedCLIP模型上,结合ROCOv2医学图像与通用示例的定性分析表明,该方法能准确捕捉组合词语对预测的协同影响。结论:本工作为多模态模型提供了直观、临床上有意义的决策解释,满足医疗领域对连贯解释的核心需求。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are demonstrating significant capabilities in medical tasks like radiology analysis, yet providing faithful and interpretable explanations remains a key consideration for their responsible deployment in clinical settings. However, existing explanation methods, such as the widely used FIxLIP framework, often struggle with the fine-grained nature of modern tokenizers. The tokenization problem fragments clinical concepts---splitting terms like "saddle embolus" into scattered, meaningless subwords---which leads to noisy, semantically incoherent cross-modal attributions. Such fragmentation also results in a combinatorial explosion of interaction possibilities, obscuring the model's true reasoning. To address this, we introduce ParseFIxLIP, an extension that incorporates the Tree-Gram Parsing into the Banzhaf interaction game used by FIxLIP. This semantically informed strategy utilizes dependency parsing trees to define explanation players by grouping related text tokens into semantically coherent units. Our smart_depth grouping strategy, merging tokens according to spaCy token dependency tree, successfully mitigates concept fragmentation, yielding substantially more interpretable cross-modal interactions by unifying complex medical concepts. Quantitatively, while baselines struggled with the high dimensionality of long captions, our parsing approach maintained statistical robustness and semantic parsimony. Qualitative analysis on BiomedCLIP, validated on medical imagery (ROCOv2) and general examples, confirms that the approach accurately captures the synergistic influence of grouped words on model predictions. In conclusion, our work offers intuitive and clinically relevant insights into VLM decision-making, fulfilling the critical need for coherent explanations in the medical domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。