解决临床多模态数据缺失与随时间演化的难题,提升疾病预测准确性。
LongMoE: Longitudinal Multimodal Learning via Trajectory-Aware Mixture-of-Experts

- 用专家混合模型动态选择适合患者的分析路径,适应不同模态缺失情况。
- 在ADNI、OASIS-3和MIMIC-IV数据集上,缺模态时性能仍优于现有方法。
- 特别适合长期随访、模态不全的临床预测任务,如阿尔茨海默病研究。
多模态临床学习对整合影像、文本及个性化健康记录至关重要,但面临两大挑战:一是模态缺失,即患者就诊时部分数据缺失;二是纵向动态性,即观察结果的诊断意义依赖于疾病随时间演变的轨迹。现有方法分别应对,或忽略时间信息,或假设模态完整。本文提出LongMoE(纵向多专家模型),统一处理这两类问题。其结合上下文感知的补全模块、捕捉非规则就诊序列中频域模式的注意力分词模块、轨迹感知编码器以及基于上下文的稀疏专家路由机制,实现患者级专家选择。在ADNI、OASIS-3和MIMIC-IV数据集上的实验表明,该模型在模态缺失或弱同时模态条件下具有更强鲁棒性,且在全模态场景下保持竞争力,为纵向感知的多模态临床学习奠定坚实基础。
原文摘要 · Abstract (English)
Multimodal clinical learning is increasingly important for integrating diverse patient data, including imaging, text, and personalised health records. However, it faces two fundamental challenges: i) modality missingness, where arbitrary subsets of modalities are unavailable at a given patient visit, ii) longitudinal dynamics, where the diagnostic significance of an observation depends on the patient's evolving disease trajectory over time. Existing methods address these challenges in isolation: missing-modality frameworks treat each visit as an independent static snapshot and discard temporal context, while longitudinal models often assume complete modality availability and degrade under systematic modality incompleteness. We propose LongMoE (Longitudinal Mixture-of-Experts), the unified framework to jointly address both challenges. LongMoE combines a context-aware imputation module with an attentional tokenization module that captures frequency-domain temporal patterns across irregular visit sequences, a trajectory-aware encoder for modeling disease progression, and context-conditioned Sparse MoE routing for patient-specific expert selection. Experiments on ADNI, OASIS-3, and MIMIC-IV show that LongMoE improves robustness under missing or weak contemporaneous modalities and remains competitive in full-modality settings, establishing a strong foundation for longitudinally-aware multimodal clinical learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。