用脑电图研究语言理解如何从声音演变为多模态体验
Aligning Brain Signals with Multimodal Speech and Vision Embeddings
- 对比wav2vec2和CLIP模型各层嵌入，找最像大脑处理层次的表示
- 发现多模态融合与逐层组合能更好匹配脑电活动模式
- 适合对神经语言建模和跨模态认知感兴趣的学者
当我们听到单词‘house’时，不仅处理声音，还会联想到墙壁、门和记忆。大脑通过多层加工构建意义，从原始声学信息逐步发展为丰富的多模态关联。受此启发，我们基于Meta先前将脑电（EEG）信号与平均wav2vec2语音嵌入对齐的研究，提出更深层问题：预训练模型的哪些层级最能反映大脑的这种分层处理过程？我们比较了两个模型的嵌入表示：wav2vec2（将声音编码为语言）和CLIP（将词语映射到图像）。利用自然语音感知过程中采集的脑电数据，通过岭回归和对比解码评估这些嵌入与大脑活动的对齐程度。测试三种策略：单层嵌入、逐层拼接、逐层求和。结果表明，结合多模态信息与层敏感表示，可能使我们更接近解码大脑如何将语言理解为一种经验，而不仅仅是声音。
原文摘要 · Abstract (English)
When we hear the word "house", we don't just process sound, we imagine walls, doors, memories. The brain builds meaning through layers, moving from raw acoustics to rich, multimodal associations. Inspired by this, we build on recent work from Meta that aligned EEG signals with averaged wav2vec2 speech embeddings, and ask a deeper question: which layers of pre-trained models best reflect this layered processing in the brain? We compare embeddings from two models: wav2vec2, which encodes sound into language, and CLIP, which maps words to images. Using EEG recorded during natural speech perception, we evaluate how these embeddings align with brain activity using ridge regression and contrastive decoding. We test three strategies: individual layers, progressive concatenation, and progressive summation. The findings suggest that combining multimodal, layer-aware representations may bring us closer to decoding how the brain understands language, not just as sound, but as experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。