用中间层特征对齐脑电信号，提升视觉解码精度
Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding
- 用深度网络中间层替代最终层做脑电与视觉特征对齐
- 在多个模型上提升22%至58%的解码准确率
- 适合脑机接口、神经表征研究者参考
神经视觉解码是脑机接口研究的核心问题，旨在重建人类视觉感知并揭示神经表征结构。当前对比学习方法通常将脑电信号与预训练视觉编码器的最终层嵌入对齐，但这些表示侧重高层语义不变性，而脑电/脑磁图信号包含多层级视觉抽象信息，可能造成表征粒度不匹配。基于大脑表征对应深度神经网络多层级的已有证据，本文提出浅层对齐（Shallow Alignment）框架，系统探索中间视觉表示作为对齐目标。跨多个基准的大量实验表明，该方法显著优于标准的最终层对齐，性能提升达22%至58%，且随预训练视觉主干模型容量增大呈现持续增长趋势。我们进一步开展系统性实证分析，揭示性能提升机制。代码已开源。
原文摘要 · Abstract (English)
Neural visual decoding is a central problem in brain-computer interface research, aiming to reconstruct human visual perception and to elucidate the structure of neural representations. Recent contrastive neural visual decoding methods commonly align neural signals with the final embeddings of pretrained vision encoders. However, such representations are optimized for high-level semantic invariance, whereas EEG/MEG signals contain information spanning multiple levels of visual abstraction, potentially creating a representational granularity mismatch. Motivated by prior evidence that brain representations correspond to multiple levels of the DNN hierarchy, we propose Shallow Alignment, a granularity-calibration framework that systematically explores intermediate visual representations as alignment targets for neural decoding. Extensive experiments across multiple benchmarks demonstrate that Shallow Alignment significantly outperforms standard final-layer alignment, with performance gains ranging from 22% to 58% across diverse vision backbones. Notably, our approach reveals a positive scaling trend in neural visual decoding, enabling decoding performance to improve consistently with the capacity of pre-trained vision backbones. We further conduct systematic empirical analyses to shed light on the mechanisms underlying the observed performance gains. Code is available at https://github.com/yangdu-neuroai/shallow-alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。