用多模态序列模型预测人脑对自然电影的响应,效果优于传统方法。
A Multimodal Seq2Seq Transformer for Predicting Brain Responses to Naturalistic Stimuli
- 用视觉、听觉和语言输入的预训练特征,通过自回归Transformer预测脑活动
- 在分布内和分布外数据上均表现优秀,尤其捕捉了刺激与神经反应的长时序结构
- 结合共享编码器与部分个体化解码器,兼顾群体共性与个体差异
Algonauts 2025挑战赛要求构建编码模型,以预测大脑对自然主义多模态电影的全脑fMRI响应。本文提出一种序列到序列的Transformer模型,基于视觉、听觉和语言输入,自回归地预测脑活动。刺激特征通过VideoMAE、HuBERT、Qwen和BridgeTower等预训练模型提取。解码器通过双交叉注意力机制融合先前脑状态与当前刺激信息,分别关注感知内容和高层叙事摘要。核心创新在于利用多模态上下文序列预测脑活动序列,从而捕捉刺激与神经反应中的长时序结构;另一创新是采用共享编码器与部分个体化解码器,既保留跨被试共性表征,又建模个体差异。模型在分布内与分布外数据上均表现优异,验证了时序感知的多模态序列建模在脑活动预测中的有效性。代码已开源:https://github.com/Angelneer926/Algonauts_challenge。
原文摘要 · Abstract (English)
The Algonauts 2025 Challenge called on the community to develop encoding models that predict whole-brain fMRI responses to naturalistic multimodal movies. In this submission, we propose a sequence-to-sequence Transformer that autoregressively predicts fMRI activity from visual, auditory, and language inputs. Stimulus features were extracted using pretrained models including VideoMAE, HuBERT, Qwen, and BridgeTower. The decoder integrates information from prior brain states and current stimuli via dual cross-attention mechanisms that attend to both perceptual information extracted from the stimulus as well as narrative information provided by high-level summaries of the content. One core innovation of our approach is the use of sequences of multimodal context to predict sequences of brain activity, enabling the model to capture long-range temporal structure in both stimuli and neural responses. Another is the combination of a shared encoder with partial subject-specific decoder, which leverages common representational structure across subjects while accounting for individual variability. Our model achieves strong performance on both in-distribution and out-of-distribution data, demonstrating the effectiveness of temporally-aware, multimodal sequence modeling for brain activity prediction. The code is available at https://github.com/Angelneer926/Algonauts_challenge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。