用脑电图解码无声说话，首次实现可解释的神经特征可视化
Decoding Covert Speech from EEG Using a Functional Areas Spatio-Temporal Transformer
- 提出基于功能区时空注意力的Transformer模型，将脑电信号转为序列编码
- 在57名受试者上验证，能区分不同单词的无声说话神经模式
- 结果可可视化，适合脑机接口与认知神经研究者参考
无声说话指在无声音或动作情况下想象说话。由于对神经发音映射理解有限及脑电信号信噪比低，从脑电图（EEG）解码无声说话极具挑战。本研究构建了大规模多语句语音EEG数据集，涵盖57名右利手母语英语受试者，每人完成五次十秒内重复同一单词的无声与有声说话任务。针对发音过程的时空特性，我们提出功能性区域时空变换器（FAST），有效将EEG信号转换为令牌并利用Transformer架构进行序列建模。结果显示，通过可视化FAST生成的激活图，可在额叶和颞叶区域清晰分辨出每个单词被无声说出时的特异性神经特征，为无声说话的神经表征提供了可解释的新见解。这是首项此类研究，为脑电图语音解码提供了可解释性证据。代码已公开于https://github.com/Jiang-Muyun/FAST
原文摘要 · Abstract (English)
Covert speech involves imagining speaking without audible sound or any movements. Decoding covert speech from electroencephalogram (EEG) is challenging due to a limited understanding of neural pronunciation mapping and the low signal-to-noise ratio of the signal. In this study, we developed a large-scale multi-utterance speech EEG dataset from 57 right-handed native English-speaking subjects, each performing covert and overt speech tasks by repeating the same word in five utterances within a ten-second duration. Given the spatio-temporal nature of the neural activation process during speech pronunciation, we developed a Functional Areas Spatio-temporal Transformer (FAST), an effective framework for converting EEG signals into tokens and utilizing transformer architecture for sequence encoding. Our results reveal distinct and interpretable speech neural features by the visualization of FAST-generated activation maps across frontal and temporal brain regions with each word being covertly spoken, providing new insights into the discriminative features of the neural representation of covert speech. This is the first report of such a study, which provides interpretable evidence for speech decoding from EEG. The code for this work has been made public at https://github.com/Jiang-Muyun/FAST
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。