arXiv:2605.24523cs.LGcs.CL2026-05

用脑电、视觉和语言三模态对齐,实现无需训练的视觉解码。

MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding

论文配图:MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding
图 1 · 摘自论文原文
  • 通过对比学习统一脑电、图像和文本表征空间。
  • 在200类零样本测试中达到54.1%准确率,显著优于基线。
  • 适用于非侵入性脑信号,可解释性强,适合神经科学与AI交叉研究者。

从脑信号中解码视觉是计算机视觉与神经科学交汇的关键挑战,需建立神经表征与视觉计算模型之间的桥梁。我们提出一种基于脑电(EEG)、视觉和文本的三模态对比学习框架,将三者表征对齐于统一潜在空间。方法分两阶段:首先在无标签数据上通过掩码重建预训练脑电编码器,学习时空规律并迁移到下游任务;其次通过对比学习联合对齐脑电、图像与大模型生成的文本描述,文本监督作为语义正则化,注入语言结构而不淹没主要脑电-图像信号。编码器融合个体适应、通道图注意力与时空卷积嵌入。在Things-EEG2 200类零样本基准上,取得54.1% Top-1与83.4% Top-5准确率,显著优于最强基线(32.4%/64.0%),配对威尔科克森检验确认显著性(p < 0.01)。在Things-MEG上验证泛化能力。分析表明紧凑嵌入几何(CN-CLIP)性能优于大模型,解码结果符合已知视觉神经生理机制。该工作是迈向鲁棒、语义化非侵入式时序神经信号视觉解码的重要一步。代码开源:https://github.com/anon-eeg/eeg_image_decoding。

原文摘要 · Abstract (English)

Visual decoding from brain signals is a key challenge at the intersection of computer vision and neuroscience, requiring methods that bridge neural representations and computational models of vision. We introduce a tri-modal contrastive framework for EEG-based visual decoding that aligns EEG, visual, and textual representations within a unified latent space. Our approach follows a two-stage design. First, we pre-train an EEG encoder via masked reconstruction on unlabeled trials, learning spatio-temporal regularities that transfer robustly to downstream tasks. Second, we jointly align EEG, image, and LLM-generated textual descriptions through contrastive learning, where text supervision acts as a semantic regularizer that injects linguistic structure into the shared space without overwhelming the primary EEG-image signal. The encoder integrates subject-specific adaptation, graph-attention over channels, and temporal-spatial convolutional embeddings. On the Things-EEG2 200-way zero-shot benchmark, our framework achieves 54.1% Top-1 and 83.4% Top-5 accuracy, substantially exceeding the strongest prior baseline (32.4% / 64.0%), with paired Wilcoxon tests confirming significance (p < 0.01) over all in-subject baselines. We validate generalization on Things-MEG. Analysis reveals that compact embedding geometries (CN-CLIP) outperform much larger backbones, and that decoding aligns with established neurophysiology of visual processing. This work is a critical step towards robust, semantically-grounded visual decoding from non-invasive temporal neural signals. The source code is publicly available in https://github.com/anon-eeg/eeg_image_decoding.

脑机接口多模态对齐零样本解码神经信号

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。