一个模型同时实现神经活动与行为的双向预测,提升脑科学研究效率。
Neural Encoding and Decoding at Scale

- 采用多任务掩码策略,交替处理神经、行为等多模态数据。
- 在83只动物数据上预训练,微调后在编码解码任务上均达顶尖表现。
- 模型自学习出脑区信息,无需额外标注,适合脑机接口研究者使用。
近期研究显示,大规模多动物模型能有效揭示神经活动与行为之间的关系。然而,现有方法仅聚焦于从行为预测神经活动(编码)或从神经活动预测行为(解码),难以捕捉二者间的双向关联。为此,我们提出一种多模态、多任务模型NEDS,实现大规模神经编码与解码同步建模。核心是新颖的多任务掩码策略,交替进行神经、行为、同模态与跨模态掩码。我们在国际脑实验室(IBL)重复位点数据集上预训练该模型,包含83只动物完成相同视觉决策任务的记录。相比其他大规模模型,当在多动物数据上预训练并微调至新动物时,NEDS在编码和解码任务中均达到当前最优性能。令人意外的是,模型学习到的嵌入表示具有涌现特性:即使未显式训练,也能高度预测每次记录对应的脑区。本工作为构建脑部基础模型迈出关键一步,实现神经活动与行为间的无缝转换。
原文摘要 · Abstract (English)
Recent work has demonstrated that large-scale, multi-animal models are powerful tools for characterizing the relationship between neural activity and behavior. Current large-scale approaches, however, focus exclusively on either predicting neural activity from behavior (encoding) or predicting behavior from neural activity (decoding), limiting their ability to capture the bidirectional relationship between neural activity and behavior. To bridge this gap, we introduce a multimodal, multi-task model that enables simultaneous Neural Encoding and Decoding at Scale (NEDS). Central to our approach is a novel multi-task-masking strategy, which alternates between neural, behavioral, within-modality, and cross-modality masking. We pretrain our method on the International Brain Laboratory (IBL) repeated site dataset, which includes recordings from 83 animals performing the same visual decision-making task. In comparison to other large-scale models, we demonstrate that NEDS achieves state-of-the-art performance for both encoding and decoding when pretrained on multi-animal data and then fine-tuned on new animals. Surprisingly, NEDS's learned embeddings exhibit emergent properties: even without explicit training, they are highly predictive of the brain regions in each recording. Altogether, our approach is a step towards a foundation model of the brain that enables seamless translation between neural activity and behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。