arXiv:2607.15380cs.CLcs.AI2026-07

用统一文本序列让大模型搞定多模态临床预测,无需定制融合结构。

Large Language Models as Unified Multimodal Learners for Clinical Prediction

  • 将所有医疗数据转为自然语言序列,直接微调大语言模型。
  • 在3个临床任务中表现媲美甚至超越专用多模态模型。
  • 适合希望简化系统架构的医疗AI研发者和临床部署团队。

电子健康记录包含自由文本的临床描述与生命体征、化验值、共病等结构化数据。然而多数临床预测系统仍依赖任务特异的融合架构,需为每项任务重新设计编码器和组合机制。本文提出更简单的替代方案:将所有患者数据(无论模态)统一转换为自然语言序列,直接端到端微调预训练语言模型,无需修改融合结构。我们在三个临床任务上评估该方法:在MIMIC-III数据集上预测院内死亡率,在德国移植中心纵向数据上预测移植物失败,以及从急救车记录中进行急诊分诊分类。对比基于编码器(ModernBERT)和解码器(Llama 3.1、Gemma、DeepSeek-R1-Qwen、Qwen3)的微调,以及已有的多模态基线,并在移植物失败任务中与临床实际使用的梯度提升模型比较。结果显示,统一文本序列化方法在三项任务中均达到或超过专用多模态基线性能,且在移植物失败预测中优于临床部署的梯度提升模型。这表明,无需专门设计融合架构的统一序列化范式,足以完成多模态临床预测,显著降低系统复杂度,同时保持甚至超越专业化设计。

原文摘要 · Abstract (English)

Electronic health records combine free-text clinical narratives with structured measurements such as vital signs, laboratory values, and comorbidities. Yet most clinical prediction systems still rely on task-specific fusion architectures, pairing dedicated encoders for each modality with learned combination mechanisms that must be re-engineered for every new task and clinical setting. We propose a simpler alternative: convert all patient data, regardless of modality, into a single natural language sequence and fine-tune a pretrained language model end-to-end, with no architectural modification for fusion. We evaluate this approach across three clinically distinct prediction tasks: in-hospital mortality on MIMIC-III, graft failure prediction using longitudinal data from a German transplant center, and emergency triage classification from ambulance records - comparing encoder-based (ModernBERT) and decoder-based (Llama 3.1, Gemma, DeepSeek-R1-Qwen, Qwen3) fine-tuning against established multimodal baselines and, for graft failure, a gradient boosting model currently used in clinical practice for post-transplant patient management. Across all three tasks, unified textual serialization matches or exceeds task-specific multimodal baselines, and outperforms the clinically deployed gradient boosting system on graft failure prediction. These results indicate that a single serialization-based paradigm, without bespoke fusion architectures, is sufficient for multimodal clinical prediction - substantially reducing system complexity while matching or exceeding specialized designs.

临床预测多模态大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。