测试医学影像分割中视觉语言模型的抗攻击能力，发现攻击导致性能大幅下降。
Adversarial Robustness Analysis of Vision-Language Models in Medical Image Segmentation
- 用适配器微调预训练视觉语言模型，用于医学图像分割。
- 对抗攻击使DSC和IoU指标显著下降，最大降幅超30%。
- 适用于医疗AI安全研究者，警示模型在真实场景中的脆弱性。
对抗攻击在计算机视觉和视觉语言模型中已得到广泛研究，但在医学图像分析领域的视觉语言分割模型（VLSMs）中仍鲜有探索。本文针对放射科、摄影和内窥镜等不同模态的二维医学图像，系统评估了微调后的VLSMs在对抗攻击下的鲁棒性。首先，利用适配器对预训练的VLSMs进行微调以适应医学图像分割任务；随后，采用投影梯度下降（PGD）和快速梯度符号法（FGSM）施加对抗攻击，量化其对模型性能的影响。结果表明，引入对抗扰动后，模型的Dice相似系数（DSC）和交并比（IoU）显著下降，最大降幅超过30%。此外，尝试寻找通用扰动但未能成功。该工作揭示了当前医学视觉语言模型在对抗攻击下的严重脆弱性，强调了在高风险医疗场景中提升模型安全性的必要性。
原文摘要 · Abstract (English)
Adversarial attacks have been fairly explored for computer vision and vision-language models. However, the avenue of adversarial attack for the vision language segmentation models (VLSMs) is still under-explored, especially for medical image analysis. Thus, we have investigated the robustness of VLSMs against adversarial attacks for 2D medical images with different modalities with radiology, photography, and endoscopy. The main idea of this project was to assess the robustness of the fine-tuned VLSMs specially in the medical domain setting to address the high risk scenario. First, we have fine-tuned pre-trained VLSMs for medical image segmentation with adapters. Then, we have employed adversarial attacks -- projected gradient descent (PGD) and fast gradient sign method (FGSM) -- on that fine-tuned model to determine its robustness against adversaries. We have reported models' performance decline to analyze the adversaries' impact. The results exhibit significant drops in the DSC and IoU scores after the introduction of these adversaries. Furthermore, we also explored universal perturbation but were not able to find for the medical images. \footnote{https://github.com/anjilab/secure-private-ai}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。