让大模型读懂脑功能连接图,实现跨数据集零样本泛化
FCN-LLM: Empower LLM for Brain Functional Connectivity Network Understanding via Graph-level Multi-task Instruction Tuning
- 用多尺度图编码器将脑网络映射到大模型语义空间
- 在19类人群属性上实现零样本泛化,优于传统方法
- 适合神经科学与临床诊断方向的研究者使用
大语言模型在语言理解与推理方面取得显著成果,其多模态扩展已能处理图像、视频和音频。受此启发,基于静息态fMRI的脑功能连接网络(FCN)基础模型在临床任务中展现出潜力。然而,现有方法未将FCN与文本模态对齐,限制了大模型直接理解FCN的能力。为此,我们提出FCN-LLM框架,通过图级多任务指令微调,使大模型能够理解FCN。该方法采用多尺度FCN编码器,捕捉脑区、功能子网及全脑特征,并将其投影至大模型语义空间。设计涵盖19类个体特异性属性(包括人口学、表型和精神疾病)的多范式指令任务。采用多阶段学习策略,先对齐FCN嵌入与大模型,再联合微调以捕获高层语义信息。在大规模多中心FCN数据库上的实验表明,FCN-LLM在未见数据集上实现强零样本泛化性能,优于传统监督模型和基础模型。本工作提出了脑功能网络与大模型融合的新范式,为神经科学提供了一个灵活且可解释的框架。
原文摘要 · Abstract (English)
Large Language Models have achieved remarkable success in language understanding and reasoning, and their multimodal extensions enable comprehension of images, video, and audio. Inspired by this, foundation models for brain functional connectivity networks derived from resting-state fMRI have shown promise in clinical tasks. However, existing methods do not align FCNs with the text modality, limiting the ability of LLMs to directly understand FCNs. To address this, we propose FCN-LLM, a framework that enables LLMs to understand FCNs through graph-level, multi-task instruction tuning. Our approach employs a multi-scale FCN encoder capturing brain-region, functional subnetwork, and whole-brain features, projecting them into the semantic space of LLM. We design multi-paradigm instruction tasks covering 19 subject-specific attributes across demographics, phenotypes, and psychiatric conditions. A multi-stage learning strategy first aligns FCN embeddings with the LLM and then jointly fine-tunes the entire model to capture high-level semantic information. Experiments on a large-scale, multi-site FCN database show that FCN-LLM achieves strong zero-shot generalization on unseen datasets, outperforming conventional supervised and foundation models. This work introduces a new paradigm for integrating brain functional networks with LLMs, offering a flexible and interpretable framework for neuroscience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。