评估CLIP模型在胸部X光中的公平性与鲁棒性,发现年龄影响大,对胸腔引流管依赖强。
Fairness and Robustness of CLIP-Based Models for Chest X-rays
- 用三组公开数据集测试六种CLIP模型的公平性与抗干扰能力
- 年龄差异导致性能差距明显,但性别种族影响较小;无引流管时表现显著下降
- 模型嵌入可被用于识别敏感属性,但主成分分析难以揭示此类模式
受CLIP在自然图像-文本领域优异表现的启发,近期研究将此类架构应用于医学任务,特别是在放射科,因存在大量图像与报告配对数据(如胸部X光)。尽管这些模型在准确性和区分能力上表现良好,其在不同临床任务中的公平性与鲁棒性仍缺乏系统评估。本研究在MIMIC-CXR、NIH-CXR14和NEATX三个公开数据集上,全面评估了六种广泛使用的CLIP基模型在胸部X光分类任务中的表现。我们基于年龄、性别和种族,评估模型在六种条件下的公平性。同时,通过对比有无胸腔引流管的气胸病例,检验模型对捷径学习的鲁棒性。结果表明,不同年龄患者间存在明显性能差异,而其他属性则表现更均衡。所有模型在无引流管图像上的性能均显著下降,表明其依赖于虚假相关。我们进一步分析了模型生成的嵌入表示:虽然敏感属性可在嵌入中被分类出,但主成分分析未显示明显模式,凸显可视化方法在评估模型局限性方面的不足。代码已开源。
原文摘要 · Abstract (English)
Motivated by the strong performance of CLIP-based models in natural image-text domains, recent efforts have adapted these architectures to medical tasks, particularly in radiology, where large paired datasets of images and reports, such as chest X-rays, are available. While these models have shown encouraging results in terms of accuracy and discriminative performance, their fairness and robustness in the different clinical tasks remain largely underexplored. In this study, we extensively evaluate six widely used CLIP-based models on chest X-ray classification using three publicly available datasets: MIMIC-CXR, NIH-CXR14, and NEATX. We assess the models fairness across six conditions and patient subgroups based on age, sex, and race. Additionally, we assess the robustness to shortcut learning by evaluating performance on pneumothorax cases with and without chest drains. Our results indicate performance gaps between patients of different ages, but more equitable results for the other attributes. Moreover, all models exhibit lower performance on images without chest drains, suggesting reliance on spurious correlations. We further complement the performance analysis with a study of the embeddings generated by the models. While the sensitive attributes could be classified from the embeddings, we do not see such patterns using PCA, showing the limitations of these visualisation techniques when assessing models. Our code is available at https://github.com/TheoSourget/clip_cxr_fairness
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。