首个从脑电波直接解码出通顺句子的模型，突破非侵入式脑机接口瓶颈。
BELT-2: Bootstrapping EEG-to-Language representation alignment for multi-task brain decoding
- 在字节对编码层面实现脑电与语言对齐，提升信号理解精度
- 在ZuCo数据集上实现52.2%的BLEU-1得分，性能提升31%-162%
- 结合大语言模型，适合脑机接口、神经科学和AI生成研究者
大型语言模型（LLMs）在多模态应用中表现卓越，但将其与人类大脑或脑动态结合仍鲜有探索。本文提出BELT-2，一种开创性的多任务模型，旨在提升从脑电图（EEG）信号中进行编码与解码的性能。为增强EEG编码器质量，BELT-2首次创新性地采用字节对编码（BPE）级的EEG-语言对齐，并在EEG域内集成多任务训练与解码。受“用GPT连接大脑”理念启发，通过在EEG编码器中间输出上使用前缀调优（prefix-tuning），将多任务EEG编码器与大语言模型相连。这些创新使BELT-2成为首个能够从非侵入性脑信号中解码出连贯可读句子的工作。实验表明，其在定量和定性指标上均显著优于先前方法，在ZuCo数据集上达到52.2%的BLEU-1得分；在其他翻译基准上性能提升达31%至162%。代码可通过匿名链接获取。
原文摘要 · Abstract (English)
The remarkable success of large language models (LLMs) across various multi-modality applications is well established. However, integrating large language models with humans, or brain dynamics, remains relatively unexplored. In this paper, we introduce BELT-2, a pioneering multi-task model designed to enhance both encoding and decoding performance from EEG signals. To bolster the quality of the EEG encoder, BELT-2 is the first work to innovatively 1) adopt byte-pair encoding (BPE)-level EEG-language alignment and 2) integrate multi-task training and decoding in the EEG domain. Inspired by the idea of \textbf{\textit{Bridging the Brain with GPT}}, we further connect the multi-task EEG encoder with LLMs by utilizing prefix-tuning on intermediary output from the EEG encoder. These innovative efforts make BELT-2 a pioneering breakthrough, making it the first work in the field capable of decoding coherent and readable sentences from non-invasive brain signals. Our experiments highlight significant advancements over prior techniques in both quantitative and qualitative measures, achieving a decoding performance with a BLEU-1 score of 52.2\% on the ZuCo dataset. Furthermore, BELT-2 shows a remarkable improvement ranging from 31\% to 162\% on other translation benchmarks. Codes can be accessed via the provided anonymous link~\footnote{https://anonymous.4open.science/r/BELT-2-0048}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。