arXiv:2608.18774cs.CVcs.LG2026-08中稿 · publication at the…

提出跨模态公平表示框架,提升皮肤疾病诊断的准确率与公平性。

MIFR: A Modality-Invariant and Fair Representation Framework for Skin Disease Classification

论文配图:MIFR: A Modality-Invariant and Fair Representation Framework for Skin Disease Classification
图 1 · 摘自论文原文
  • 融合临床与皮肤镜图像,通过多任务损失实现跨模态对齐。
  • 在多个数据集上表现优于基线模型,肤色差异下的分类公平性显著提升。
  • 适合医学图像分析、AI辅助诊疗及公平性研究者参考。

皮肤疾病是全球重大公共卫生问题,但现有机器学习诊断工具存在仅依赖单一模态和不同肤色间性能偏差两大缺陷。本文提出一种模态不变且公平的表示框架(MIFR),将临床照片与皮肤镜图像配对,采用ViT编码器分别提取特征,并通过模态专属投影头映射至高维嵌入空间。模型使用五组件多目标损失训练:分类用加权交叉熵,公平性由混淆矩阵与肤色分类损失保障,模态内对齐通过监督对比损失实现,模态不变性则由跨模态对齐损失约束。在HIBA+Derm7pt、PAD-UFES-20和ISIC 2019三个数据集上的实验表明,该方法在内部数据集上具备竞争力的预测性能与公平性;t-SNE可视化证实同病种的临床与皮肤镜嵌入在几何上对齐,验证了联合目标的有效性。

原文摘要 · Abstract (English)

Skin diseases represent a major global public health burden, yet machine learning tools developed to assist in their diagnosis suffer from two critical limitations: reliance on only one modality for diagnosis and systematic performance disparities across skin tones. While existing approaches address each challenge separately, this work proposes a modality-invariant framework with fair representation (MIFR) for skin disease classification. The architecture pairs clinical photographs with dermoscopic images using ViT-based encoders, projecting each input into a high-dimensional embedding space via modality-specific projection heads. The resulting model is trained with a five-component multi-objective loss including weighted cross-entropy for classification, confusion and skin-type classification losses for fairness, per-modality supervised contrastive loss for class alignment, and a modality-invariance loss for clinical and dermoscopic modality alignment. Experiments on the HIBA+Derm7pt paired dataset and the external PAD-UFES-20 and ISIC 2019 datasets showed that modality-invariant representation learning provides competitive predictive performance compare to relevant baseline models and competitive fairness on the internal dataset. t-SNE visualizations confirmed that clinical and dermoscopic embeddings of the same disease are geometrically aligned, validating the joint objectives.

皮肤疾病跨模态公平性ViT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。