轻量多模态模型,用少量数据提升脑科学分析效率
BrainSymphony: A parameter-efficient multimodal foundation model for brain dynamics with limited data
- 并行处理fMRI时空数据,用图注意力编码脑结构连接
- 在多个任务上超越更大模型,仅需少量数据训练
- 适合临床与神经科学研究者快速部署与解释结果
基础模型正改变神经科学,但常体积庞大、依赖大量数据且难以部署。本文提出BrainSymphony,一种轻量级、参数高效的多模态基础模型,可无缝集成fMRI时间序列与扩散磁共振衍生的结构连接信息,支持单模态或双模态训练与推理,无需修改架构,且所需数据远少于现有最优方法。模型通过并行的空间与时间变压器流处理fMRI数据,经由Perceiver模块压缩为紧凑嵌入;同时采用新型带符号图变压器编码扩散MRI的解剖连接。两种互补表征通过自适应融合机制整合。尽管设计紧凑,BrainSymphony在预测、分类和无监督网络发现等基准测试中持续优于更大模型。其泛化性与可解释性体现在:在独立的裸盖菇素神经影像数据集中,注意力图揭示了药物诱导的皮层层级动态重组。BrainSymphony实现了可访问、可解释、临床相关的成果,表明经过架构设计的多模态模型可超越庞大模型,推动人工智能在神经科学中的应用。
原文摘要 · Abstract (English)
Foundation models are transforming neuroscience but are often prohibitively large, data-hungry, and difficult to deploy. Here, we introduce BrainSymphony, a lightweight and parameter-efficient foundation model with plug-and-play integration of fMRI time series and diffusion-derived structural connectivity, allowing unimodal or multimodal training and deployment without architectural changes while requiring substantially less data compared to the state-of-the-art. The model processes fMRI time series through parallel spatial and temporal transformer streams, distilled into compact embeddings by a Perceiver module, while a novel signed graph transformer encodes anatomical connectivity from diffusion MRI. These complementary representations are then combined through an adaptive fusion mechanism. Despite its compact design, BrainSymphony consistently outperforms larger models on benchmarks spanning prediction, classification, and unsupervised network discovery. Highlighting the model's generalizability and interpretability, attention maps reveal drug-induced context-dependent reorganization of cortical hierarchies in an independent psilocybin neuroimaging dataset. BrainSymphony delivers accessible, interpretable, and clinically meaningful results and demonstrates that architecturally informed, multimodal models can surpass much larger counterparts and advance applications of AI in neuroscience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。