arXiv:2608.25970cs.CVcs.AI2026-08

用原型对齐方法,让模型在无辅助数据时也能利用不完整多模态数据提升预测能力。

PANDA - Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimers MRI and TCGA Pathology

论文配图:PANDA - Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimers MRI and TCGA Pathology
图 1 · 摘自论文原文
  • 通过原型锚定实现跨模态对齐,无需辅助数据即可迁移信息
  • 在阿尔茨海默病分类中,准确率提升7.9个百分点,假阳性降低24.3个百分点
  • 适用于零重叠数据场景,适合临床部署中数据不全的医疗应用

多模态医学预测常面临数据配对不全问题:部分受试者有辅助模态数据(如影像、基因),但部署时无法获取。本文提出PANDA(原型锚定对齐)框架,分两阶段训练:第一阶段从有配对的数据中学习共享嵌入并估计类别原型;第二阶段在所有受试者上训练主模态编码器,使用交叉熵损失和对冻结原型的对齐约束。因监督基于类别原型,该方法可适应任意配对率,包括零重叠。在1021例多扫描仪ADNI队列中,以MRI为主模态,辅以表型评分(44.8%配对)、FDG-PET(18.7%)及外部手写运动学(0%重叠),相比仅用MRI的基线,PANDA将AUC提升至0.868±0.020(+7.9pp),1.5T扫描下健康对照假阳性降低24.3pp;在可训练的Conv5-FC3架构上达到AUC 0.893(最佳表现)。配对率消融实验表明,联合原型在5%~75%配对率下仍稳定。在TCGA-Lung生存预测任务中,以全切片图像为输入,RNA-seq为辅助数据,PANDA在2年总生存率预测中AUC提升3.5pp,Cox PH模型的C-index提高9.0点,优于全融合训练(后者反而劣于仅用WSI),且推理时不需RNA数据;小样本导致置信区间较宽,效果未达传统显著性。总体而言,PANDA提供了一种面向实际部署的机制,可在辅助数据缺失情况下有效利用不完整多模态信息。

原文摘要 · Abstract (English)

Multimodal medical prediction often faces incomplete pairing: auxiliary modalities with complementary signal are available for only a subset of subjects (or none) and cannot be assumed at deployment. We introduce PANDA (Prototype Anchored Data Alignment), a two-stage framework that transfers auxiliary information to a primary-modality model without auxiliary inputs at inference. Stage 1 learns a shared embedding from the paired subset and estimates class prototypes from auxiliary modalities; Stage 2 trains the primary encoder on all subjects using cross-entropy plus alignment to the frozen prototypes. Because supervision is defined at the class-prototype level, PANDA accommodates arbitrary pairing rates, including zero subject overlap. We evaluate PANDA on two applications. On a 1,021-subject multi-scanner ADNI cohort, we perform AD/CN classification with three auxiliary modalities at distinct pairing rates: tabular scores (44.8%), FDG-PET (18.7%), and external handwriting kinematics (0% overlap). Relative to the same-backbone MRI-only baseline, PANDA attains AUC 0.868 +-0.020 (+7.9pp) and reduces 1.5T CN false positives by 24.3pp; on a fully trainable Conv5-FC3 backbone it reaches AUC 0.893 (best overall). A pairing-rate ablation shows that the joint anchor remains within seed noise from 75% to 5% pairing. On TCGA-Lung survival prediction from whole-slide images with RNA-seq as auxiliary data, PANDA improves over WSI-only on 2-year OS (AUC +3.5pp) and Cox PH (C-index +9.0pts) and outperforms full-fusion training, which underperforms WSI-only, while requiring no RNA at inference; wide confidence intervals on this smaller cohort keep the gains below conventional significance. Overall, PANDA provides a deployment-oriented mechanism for leveraging incomplete auxiliary modalities to improve primary-modality prediction.

多模态学习医疗影像零重叠原型对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。