跨模态脑活动翻译框架，统一不同脑成像数据的时空表示。
Latent Representation Learning for Multimodal Brain Activity Translation
- 用注意力小波分解、图注意力网络和循环层建模多模态脑信号
- 在隐层表征中成功分类外部刺激，验证了表征有效性
- 适合神经科学与临床研究者用于多模态脑数据融合
神经科学使用多种神经影像技术，如具有高时间分辨率的脑电图（EEG）和具有更高空间精度的血流动力学模态（fMRI），但整合这些异构数据仍具挑战，限制了对脑功能的全面理解。我们提出时空多模态脑活动对齐框架（SAMBA），通过学习一个无模态特异性偏差的统一潜在空间，弥合不同模态间的时空分辨率差距。SAMBA引入基于注意力的小波分解进行电生理信号频谱滤波，采用图注意力网络建模功能脑单元间的功能连接，并利用循环层捕捉脑信号的时间自相关性。实验表明，SAMBA训练不仅实现跨模态翻译，还学习到丰富的脑信息处理表征。我们进一步展示，可基于SAMBA隐藏层学到的表征准确分类引发脑活动的外部刺激，为神经科学研究和临床应用提供广泛前景。
原文摘要 · Abstract (English)
Neuroscience employs diverse neuroimaging techniques, each offering distinct insights into brain activity, from electrophysiological recordings such as EEG, which have high temporal resolution, to hemodynamic modalities such as fMRI, which have increased spatial precision. However, integrating these heterogeneous data sources remains a challenge, which limits a comprehensive understanding of brain function. We present the Spatiotemporal Alignment of Multimodal Brain Activity (SAMBA) framework, which bridges the spatial and temporal resolution gaps across modalities by learning a unified latent space free of modality-specific biases. SAMBA introduces a novel attention-based wavelet decomposition for spectral filtering of electrophysiological recordings, graph attention networks to model functional connectivity between functional brain units, and recurrent layers to capture temporal autocorrelations in brain signal. We show that the training of SAMBA, aside from achieving translation, also learns a rich representation of brain information processing. We showcase this classify external stimuli driving brain activity from the representation learned in hidden layers of SAMBA, paving the way for broad downstream applications in neuroscience research and clinical contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。