用fMRI信号作提示，让GPT-2解码大脑听觉信息
BP-GPT: Auditory Neural Decoding Using fMRI-prompted LLM
- 将fMRI提取的脑信号作为提示输入GPT-2进行文本解码
- 在公开数据集上实现METEOR提升4.61、BERTScore提升2.43
- 首次实现全端到端的脑信号到语言的生成，适合神经解码研究者
从脑信号中解码语言信息是脑机接口领域的重要方向，尤其在解析fMRI信号的语义信息方面。现有方法虽使用大语言模型（LLM）实现该目标，但未采用端到端方式，且在fMRI到文本映射中回避了LLM的作用，限制了其在听觉解码中的应用。本文提出一种新方法——脑提示GPT（BP-GPT），通过将fMRI提取的脑表征作为提示，驱动GPT-2将脑信号还原为刺激文本。同时引入文本提示，对齐脑提示与语义空间，使模型能提取更鲁棒的脑提示，提升预训练语言模型的解码能力。我们在开源听觉语义解码数据集上评估了该方法，相比当前最优方法，在所有受试者上实现了最高4.61的METEOR得分和2.43的BERTScore提升。实验表明，以脑表征为提示进一步驱动大语言模型进行听觉神经解码是可行且有效的。代码已开源：https://github.com/1994cxy/BP-GPT。
原文摘要 · Abstract (English)
Decoding language information from brain signals represents a vital research area within brain-computer interfaces, particularly in the context of deciphering the semantic information from the fMRI signal. Although existing work uses LLM to achieve this goal, their method does not use an end-to-end approach and avoids the LLM in the mapping of fMRI-to-text, leaving space for the exploration of the LLM in auditory decoding. In this paper, we introduce a novel method, the Brain Prompt GPT (BP-GPT). By using the brain representation that is extracted from the fMRI as a prompt, our method can utilize GPT-2 to decode fMRI signals into stimulus text. Further, we introduce the text prompt and align the fMRI prompt to it. By introducing the text prompt, our BP-GPT can extract a more robust brain prompt and promote the decoding of pre-trained LLM. We evaluate our BP-GPT on the open-source auditory semantic decoding dataset and achieve a significant improvement up to 4.61 on METEOR and 2.43 on BERTScore across all the subjects compared to the state-of-the-art method. The experimental results demonstrate that using brain representation as a prompt to further drive LLM for auditory neural decoding is feasible and effective. The code is available at https://github.com/1994cxy/BP-GPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。