arXiv:2412.17829eess.SPcs.LG2024-12被引 16

用723人数据训练模型,从脑电信号中解码单词,跨语言跨任务表现优异。

Decoding individual words from non-invasive brain recordings across 723 participants

  • 构建新深度学习框架,从头皮电/磁信号中解码单个词。
  • 在500万词数据上测试,对未见词汇也能准确解码。
  • 发现阅读比听觉易解码,数据量越多效果越好。

深度学习已实现少数植入电极者脑内语言的解码,但非侵入性记录(如EEG和MEG)仍具挑战。本文提出一种新型深度学习流水线,基于723名参与者、五百万词(英、法、荷语)的书写或口语输入数据进行训练与评估。模型在不同参与者、设备、语言和任务中均优于现有方法,且可解码训练集外词汇。分析表明:MEG和阅读条件更易解码;每参与者多收集数据优于跨多人重复刺激。解码性能随训练数据量和测试时平均次数增加而提升。单词预测显示模型不仅依赖语义,还捕捉词性、长度、字母等表面特征,尤其在阅读条件下显著。研究明确了非侵入性自然语言脑解码的发展路径与现存挑战。

原文摘要 · Abstract (English)

Deep learning has recently enabled the decoding of language from the neural activity of a few participants with electrodes implanted inside their brain. However, reliably decoding words from non-invasive recordings remains an open challenge. To tackle this issue, we introduce a novel deep learning pipeline to decode individual words from non-invasive electro- (EEG) and magneto-encephalography (MEG) signals. We train and evaluate our approach on an unprecedentedly large number of participants (723) exposed to five million words either written or spoken in English, French or Dutch. Our model outperforms existing methods consistently across participants, devices, languages, and tasks, and can decode words absent from the training set. Our analyses highlight the importance of the recording device and experimental protocol: MEG and reading are easier to decode than EEG and listening, respectively, and it is preferable to collect a large amount of data per participant than to repeat stimuli across a large number of participants. Furthermore, decoding performance consistently increases with the amount of (i) data used for training and (ii) data used for averaging during testing. Finally, single-word predictions show that our model effectively relies on word semantics but also captures syntactic and surface properties such as part-of-speech, word length and even individual letters, especially in the reading condition. Overall, our findings delineate the path and remaining challenges towards building non-invasive brain decoders for natural language.

脑机接口语言解码深度学习非侵入

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。