融合视觉语言与大模型,精准预测慢性病风险。
Multimodal Health Risk Prediction System for Chronic Diseases via Vision-Language Fusion and Large Language Models
- 分层堆叠的视觉语言Transformer,结合大模型推理头。
- 在MIMIC-IV数据集上达到0.90平均AUROC,校准误差仅2.7%。
- 适合医疗AI研究者与临床风险评估系统开发者。
随着慢性病负担加重及临床数据多模态、异构化(如医学影像、自由文本记录、可穿戴设备流数据等),亟需统一的多模态AI框架实现个体健康风险的主动预测。我们提出VL-RiskFormer,一种分层堆叠的视觉-语言多模态Transformer,其顶层嵌入大语言模型(LLM)推理头。该系统基于现有视觉-语言模型(如PaLM-E、LLaVA)的双流架构,具有四项创新:(i) 利用动量更新编码器和去偏置InfoNCE损失,对放射影像、眼底图与可穿戴设备照片与对应临床叙述进行跨模态对比预训练及细粒度对齐;(ii) 采用时间融合模块,通过自适应时间间隔位置编码将不规则就诊序列整合进因果Transformer解码器;(iii) 设计疾病本体图适配器,将ICD-10编码注入视觉与文本通道,并借助图注意力机制推断共病模式。在MIMIC-IV纵向队列数据集上,VL-RiskFormer实现平均AUROC 0.90,预期校准误差为2.7%。
原文摘要 · Abstract (English)
With the rising global burden of chronic diseases and the multimodal and heterogeneous clinical data (medical imaging, free-text recordings, wearable sensor streams, etc.), there is an urgent need for a unified multimodal AI framework that can proactively predict individual health risks. We propose VL-RiskFormer, a hierarchical stacked visual-language multimodal Transformer with a large language model (LLM) inference head embedded in its top layer. The system builds on the dual-stream architecture of existing visual-linguistic models (e.g., PaLM-E, LLaVA) with four key innovations: (i) pre-training with cross-modal comparison and fine-grained alignment of radiological images, fundus maps, and wearable device photos with corresponding clinical narratives using momentum update encoders and debiased InfoNCE losses; (ii) a time fusion block that integrates irregular visit sequences into the causal Transformer decoder through adaptive time interval position coding; (iii) a disease ontology map adapter that injects ICD-10 codes into visual and textual channels in layers and infers comorbid patterns with the help of a graph attention mechanism. On the MIMIC-IV longitudinal cohort, VL-RiskFormer achieved an average AUROC of 0.90 with an expected calibration error of 2.7 percent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。