arXiv:2604.27343cs.CV2026-04被引 2

融合影像与患者信息,动态优化多模态皮肤病变分类。

JI-ADF: Joint-Individual Learning with Adaptive Decision Fusion for Multimodal Skin Lesion Classification

论文配图:JI-ADF: Joint-Individual Learning with Adaptive Decision Fusion for Multimodal Skin Lesion Classification
图 1 · 摘自论文原文
  • 联合学习+自适应融合,动态调整不同模态贡献。
  • 在MILK10k数据集上提升敏感度与骰子系数。
  • 适合临床部署,可解释性强,应对真实医疗场景。

皮肤病变分类对早期皮肤病诊断至关重要,但现有辅助系统多依赖皮肤镜图像,忽视临床中常见的多模态证据。为此,我们提出JI-ADF,一种融合皮肤镜图像、临床照片和结构化患者元数据的三模态深度学习框架。该架构结合联合多模态表征学习、模态专属辅助监督及基于样本的自适应决策融合机制,动态校准各模态贡献。为增强跨模态推理并保留模态特异性信息,引入多模态融合注意力(MMFA)模块。在反映真实临床采集条件且存在严重类别不平衡的大规模MILK10k基准上评估,所提方法在各类病变上表现稳健均衡,提升敏感度与骰子分数,同时保持高特异性和良好校准性。大量分析包括模态消融、校准评估及Grad-CAM可视化,进一步验证模型鲁棒性与临床可解释性。结果表明,JI-ADF为真实临床环境下的多模态皮肤病变分类提供了可靠且实用的基础。

原文摘要 · Abstract (English)

Skin lesion classification is essential for early dermatological diagnosis, yet many existing computer-aided systems rely primarily on dermoscopic images and underutilize the multimodal evidence routinely available in clinical practice. To address this gap, we propose \textbf{JI-ADF}, a trimodal deep learning framework that integrates dermoscopic images, clinical photographs, and structured patient metadata for clinically grounded skin lesion classification. The proposed architecture combines joint multimodal representation learning with modality-specific auxiliary supervision and an adaptive decision fusion mechanism that dynamically calibrates modality contributions on a per-sample basis. To enhance cross-modal reasoning while preserving modality-specific evidence, we further introduce a multimodal fusion attention (MMFA) module. We evaluate JI-ADF on the large-scale MILK10k benchmark, which reflects real-world clinical acquisition conditions and severe class imbalance. The proposed method demonstrates strong and well-balanced performance across lesion categories, improving sensitivity and Dice score while maintaining high specificity and good calibration. Extensive analyses, including modality ablation, calibration evaluation, and Grad-CAM visualization, further confirm the robustness and clinically meaningful behavior of the model. These results indicate that JI-ADF provides a reliable and practical foundation for multimodal skin lesion classification in real-world clinical settings.

皮肤病变多模态临床应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。