用对比学习对齐脑电、脑磁和核磁数据，实现视觉信息解码与跨模态转换。
Towards Neural Foundation Models for Vision: Aligning EEG, MEG, and fMRI Representations for Decoding, Encoding, and Modality Conversion
- 通过对比学习对齐EEG、MEG、fMRI三类脑数据的语义表示。
- 在解码、编码和模态转换任务中均实现高精度语义还原。
- 为脑机接口与神经科学提供统一的多模态基础模型，适合相关研究者。
本文提出一种新方法，构建用于对齐脑活动多模态表征与视觉刺激的基础模型，利用对比学习融合脑电（EEG）、脑磁（MEG）和功能磁共振成像（fMRI）数据。通过三个关键实验验证：从神经数据中解码视觉信息、将图像编码为神经表征、在不同神经模态间进行转换。结果表明，该模型能准确捕捉跨多种脑成像技术的语义信息，展现出在解码、编码及模态转换任务中的潜力。
原文摘要 · Abstract (English)
This paper presents a novel approach towards creating a foundational model for aligning neural data and visual stimuli across multimodal representationsof brain activity by leveraging contrastive learning. We used electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) data. Our framework's capabilities are demonstrated through three key experiments: decoding visual information from neural data, encoding images into neural representations, and converting between neural modalities. The results highlight the model's ability to accurately capture semantic information across different brain imaging techniques, illustrating its potential in decoding, encoding, and modality conversion tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。