用大模型统一脑电、脑磁和脑血氧信号,实现跨模态脑活动解码
One Brain, Omni Modalities: Towards Unified Non-Invasive Brain Decoding with Large Language Models
- 构建统一编码器融合脑电/脑磁与功能磁共振信号
- 跨模态融合使解码准确率高于单一模态基线
- 支持刺激感知解码,可验证感官信号与脑响应的因果关系
通过非侵入式记录解码脑功能需整合高频电磁信号(EEG/MEG)与低频代谢信号(fMRI)。尽管来源相同,但两者长期被分割在独立分析流程中,限制了对脑活动的整体理解。为此,我们提出NOBEL——一种神经-全模态脑编码大语言模型,将不同模态信号统一映射到大模型语义嵌入空间。其架构采用统一编码器处理EEG/MEG,创新设计双路径策略处理fMRI,将脑信号与外部感官刺激对齐至共享标记空间,并以大语言模型作为通用主干。大量实验表明,NOBEL在标准单模态任务中表现稳健,且电磁与代谢信号融合后解码精度显著优于单模态基线,验证了多模态互补性。此外,该模型在NSD与HAD数据集上实现了多被试fMRI数据的刺激感知解码,能有效解析视觉语义,并通过直接输入刺激验证感官信号与神经反应间的因果联系。研究推动了非侵入式脑解码的统一,展现了全模态脑理解的潜力。
原文摘要 · Abstract (English)
Deciphering brain function through non-invasive recordings requires synthesizing complementary high-frequency electromagnetic (EEG/MEG) and low-frequency metabolic (fMRI) signals. However, despite their shared neural origins, extreme discrepancies have traditionally confined these modalities to isolated analysis pipelines, hindering a holistic interpretation of brain activity. To bridge this fragmentation, we introduce \textbf{NOBEL}, a \textbf{n}euro-\textbf{o}mni-modal \textbf{b}rain-\textbf{e}ncoding \textbf{l}arge language model (LLM) that unifies these heterogeneous signals within the LLM's semantic embedding space. Our architecture integrates a unified encoder for EEG and MEG with a novel dual-path strategy for fMRI, aligning non-invasive brain signals and external sensory stimuli into a shared token space, then leverages an LLM as a universal backbone. Extensive evaluations demonstrate that NOBEL serves as a robust generalist across standard single-modal tasks. We also show that the synergistic fusion of electromagnetic and metabolic signals yields higher decoding accuracy than unimodal baselines, validating the complementary nature of multiple neural modalities. Furthermore, NOBEL exhibits strong capabilities in stimulus-aware decoding, effectively interpreting visual semantics from multi-subject fMRI data on the NSD and HAD datasets while uniquely leveraging direct stimulus inputs to verify causal links between sensory signals and neural responses. NOBEL thus takes a step towards unifying non-invasive brain decoding, demonstrating the promising potential of omni-modal brain understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。