用1.5万份脑电与病历训练跨模态模型，实现少样本临床表型分析
EEG-Language Pretraining for Highly Label-Efficient Clinical Phenotyping
- 结合时序裁剪与文本分段，用多实例学习缓解脑电与病历错配
- 在4项临床评估中显著优于仅用脑电的模型，首次实现零样本分类与检索
- 适合医疗AI研究者，尤其关注少标注数据场景下的脑电分析
多模态语言建模已在表示学习中取得突破，但在功能脑数据用于临床表型分析方面仍属空白。本文首次提出基于临床报告和15,000例脑电图（EEG）的脑电-语言模型（ELM）。通过在这一新领域引入多模态对齐，并结合时序裁剪与文本分段，基于多实例学习扩展方法，有效缓解了无关脑电或文本片段间的错配问题。所提出的多模态模型在四项临床评估中均显著优于仅使用脑电的模型，首次实现零样本分类及神经信号与病历的联合检索。结果表明，ELM具有显著临床应用潜力，标志着该方向的重要进展。
原文摘要 · Abstract (English)
Multimodal language modeling has enabled breakthroughs for representation learning, yet remains unexplored in the realm of functional brain data for clinical phenotyping. This paper pioneers EEG-language models (ELMs) trained on clinical reports and 15000 EEGs. We propose to combine multimodal alignment in this novel domain with timeseries cropping and text segmentation, enabling an extension based on multiple instance learning to alleviate misalignment between irrelevant EEG or text segments. Our multimodal models significantly improve over EEG-only models across four clinical evaluations and for the first time enable zero-shot classification as well as retrieval of both neural signals and reports. In sum, these results highlight the potential of ELMs, representing significant progress for clinical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。