提出新理论提升图文模型可解释性,效果显著且更快。
Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability
- 重构信息瓶颈理论,更好满足可解释性要求。
- 图文可解释性分别提升9%和58.83%,速度加快63.95%。
- 适合关注模型安全、医疗等高风险场景的研究者。
识别多模态图像-文本表示的任务日益受到关注,尤其以CLIP(对比语言-图像预训练)为代表模型在学习图像与文本间复杂关联方面表现出色。尽管如此,确保此类模型的可解释性对其实现安全部署至关重要,如在医疗领域。现有可解释性方法多针对单模态任务,难以有效迁移至多模态场景,因表征结构差异。信息瓶颈方法虽被用于提升CLIP可解释性,但常受限于强假设或内在随机性。为此,本文提出窄化信息瓶颈理论(Narrowing Information Bottleneck Theory),从根本上重新定义传统瓶颈方法,专为满足现代归因公理而设计,提供更稳健可靠的多模态模型可解释性解决方案。实验表明,相比最先进方法,本方法平均提升图像可解释性9%、文本可解释性58.83%,处理速度提升63.95%。代码已公开于https://github.com/LMBTough/NIB。
原文摘要 · Abstract (English)
The task of identifying multimodal image-text representations has garnered increasing attention, particularly with models such as CLIP (Contrastive Language-Image Pretraining), which demonstrate exceptional performance in learning complex associations between images and text. Despite these advancements, ensuring the interpretability of such models is paramount for their safe deployment in real-world applications, such as healthcare. While numerous interpretability methods have been developed for unimodal tasks, these approaches often fail to transfer effectively to multimodal contexts due to inherent differences in the representation structures. Bottleneck methods, well-established in information theory, have been applied to enhance CLIP's interpretability. However, they are often hindered by strong assumptions or intrinsic randomness. To overcome these challenges, we propose the Narrowing Information Bottleneck Theory, a novel framework that fundamentally redefines the traditional bottleneck approach. This theory is specifically designed to satisfy contemporary attribution axioms, providing a more robust and reliable solution for improving the interpretability of multimodal models. In our experiments, compared to state-of-the-art methods, our approach enhances image interpretability by an average of 9%, text interpretability by an average of 58.83%, and accelerates processing speed by 63.95%. Our code is publicly accessible at https://github.com/LMBTough/NIB.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。