让医学影像和文本各自独立诊断，避免模型只依赖简单信息。
UniMod: Enhancing Multi-Modal Medical Diagnosis through Cross-Modality and Within-Modality Alignment
- 图像和文本分别独立预测疾病，强制学习关键特征。
- 在哈佛青光眼数据集上达0.850 AUC，比现有方法高1.6%-1.8%。
- 无需改结构即可扩展到多标签诊断，提升均值AUC 0.097。
将医学影像与临床文本结合进行疾病诊断的多模态学习具有潜力。然而，标准多模态训练易导致捷径学习：模型仅依赖更简单的模态（如文本中的诊断线索），忽略较难提取的特征（如细微的视觉模式）。本文提出UniMod框架，通过要求每种模态独立完成诊断任务来缓解此问题。该框架同时监督图像仅、文本仅及多模态分类，促使各模态均提取诊断性特征。引入跨模态对齐以实现知识迁移，并在同诊断患者间进行模态内监督对比对齐。在Harvard-Glaucoma数据集上，UniMod达到0.850 AUC，优于OGM-GE与Gradient Blending 1.6-1.8%；在CheXpert Plus上达0.966 AUC，超越它们超5%。该方法还可无需架构调整扩展至五类多标签诊断，平均AUC提升0.097，优于CGGM。
原文摘要 · Abstract (English)
Multi-modal learning combining medical images and clinical text is promising for disease diagnosis. However, standard multi-modal training leads to shortcut learning: models exploit the easier modality (e.g., diagnostic cues in text) while neglecting harder-to-learn features (e.g., subtle visual patterns). We propose UniMod, a framework that mitigates shortcut learning by requiring each modality to predict the diagnosis on its own. It supervises image-only, text-only, and multi-modal classification simultaneously, so each modality must extract diagnostic features. We add cross-modality alignment for knowledge transfer and within-modality supervised contrastive alignment over same-diagnosis patients. On Harvard-Glaucoma, UniMod reaches 0.850 AUC, outperforming OGM-GE and Gradient Blending by 1.6-1.8%; on CheXpert Plus, it reaches 0.966 AUC, surpassing them by over 5%. UniMod also extends to 5-class multi-label diagnosis without architectural change, improving mean AUC by 0.097 over CGGM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。