提出医学影像多模态模型的链式分布攻击框架,揭示真实临床流程中的脆弱性。
CoDA: Exploring Chain-of-Distribution Attacks and Post-Hoc Token-Space Repair for Medical Vision-Language Models
- 构建链式分布攻击框架CoDA,模拟临床成像全流程的统计偏移
- 多模态模型在链式攻击下零样本性能显著下降,单阶段攻击不如组合严重
- 提出后处理修复方法,通过令牌空间对齐提升模型鲁棒性,适合临床部署
医学视觉语言模型(MVLMs)日益被用作放射科工作流的感知主干或多模态助手的视觉前端,但其在真实临床流程中的可靠性尚未充分探索。以往的鲁棒性评估通常假设输入干净且经过筛选,或仅研究单一退化,忽略了成像采集、重建、显示与传输等常规操作虽保持临床可读性却改变图像统计分布的现象。为此,我们提出CoDA,一种链式分布攻击框架,通过组合采集级阴影、重建与显示重映射、交付与导出退化等阶段,构建临床合理的图像分布偏移。在掩码结构相似性约束下,CoDA联合优化各阶段组成与参数,诱导模型失效同时保持视觉合理性。在脑部MRI、胸部X光和腹部CT上,CoDA显著降低基于CLIP的MVLM零样本性能,链式组合攻击效果始终强于单阶段攻击。我们还评估了多模态大模型作为成像真实性和质量的技术审计者,而非病理诊断者。专有大模型在CoDA扰动样本上审计可靠性下降,持续出现高置信度错误;而测试的医疗专用多模态模型在医学图像质量审计方面表现明显不足。最后,我们引入一种基于教师引导的令牌空间自适应后处理修复策略,通过局部块对齐提升对存档CoDA输出的准确率。总体而言,我们的发现刻画了医学视觉语言模型部署中的临床基础威胁面,并表明轻量级对齐可提升实际部署下的鲁棒性。
原文摘要 · Abstract (English)
Medical vision--language models (MVLMs) are increasingly used as perceptual backbones in radiology pipelines and as the visual front end of multimodal assistants, yet their reliability under real clinical workflows remains underexplored. Prior robustness evaluations often assume clean, curated inputs or study isolated corruptions, overlooking routine acquisition, reconstruction, display, and delivery operations that preserve clinical readability while shifting image statistics. To address this gap, we propose CoDA, a chain-of-distribution framework that constructs clinically plausible pipeline shifts by composing acquisition-like shading, reconstruction and display remapping, and delivery and export degradations. Under masked structural-similarity constraints, CoDA jointly optimizes stage compositions and parameters to induce failures while preserving visual plausibility. Across brain MRI, chest X-ray, and abdominal CT, CoDA substantially degrades the zero-shot performance of CLIP-style MVLMs, with chained compositions consistently more damaging than any single stage. We also evaluate multimodal large language models (MLLMs) as technical-authenticity auditors of imaging realism and quality rather than pathology. Proprietary multimodal models show degraded auditing reliability and persistent high-confidence errors on CoDA-shifted samples, while the medical-specific MLLMs we test exhibit clear deficiencies in medical image quality auditing. Finally, we introduce a post-hoc repair strategy based on teacher-guided token-space adaptation with patch-level alignment, which improves accuracy on archived CoDA outputs. Overall, our findings characterize a clinically grounded threat surface for MVLM deployment and show that lightweight alignment improves robustness in deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。