用检索增强生成技术,从脑电波中解码出有意义的句子。
RAG-based EEG-to-Text Translation Using Deep Learning and LLMs

- 结合脑电信号编码、向量检索与大模型,实现端到端句级翻译
- 在9名受试者上达到0.181的平均余弦相似度,比基线高30.45%
- 无需真实标签即可推理,适合脑机接口与自然语言理解研究
从脑电图(EEG)信号中解码语言信息仍是脑机接口领域的重大挑战,尤其在句级解码时,由于信号噪声比低而尤为困难。以往研究通常在推理阶段使用教师强制才能超越随机基线。本文提出一种基于检索增强生成(RAG)的句级脑电转文本框架,包含与语义句子嵌入对齐的EEG编码器、向量检索模块和大语言模型(LLM),用于将检索到的句子优化为连贯输出。实验基于苏黎世认知语言处理语料库(ZuCo)数据集,该数据集包含静默阅读时采集的单次试验脑电信号。为评估系统是否提取了有效信息,结果与随机基线进行对比。在9名受试者中,所提方法的平均余弦相似度为0.181 ± 0.022,优于基线的0.139 ± 0.029,相对提升达30.45%。统计分析表明该提升显著,且采用严格评估流程,推理阶段不访问真实标签。
原文摘要 · Abstract (English)
The decoding of linguistic information from electroencephalography (EEG) signals remains an extremely challenging problem in brain-computer interface (BCI) research. In particular, sentence-level decoding from EEG is difficult due to the low signal-to-noise ratio of these recordings. Previous studies tackling this problem have typically failed to surpass random baseline performance unless teacher forcing is used during the inference phase. In this work, we propose a retrieval-augmented generation (RAG)-based sentence-level EEG-to-text decoding pipeline that combines an EEG encoder aligned with semantic sentence embeddings, a vector retrieval stage, and a large language model (LLM) to refine retrieved sentences into coherent output. Experiments are conducted on the Zurich Cognitive Language Processing Corpus (ZuCo) dataset, which contains single-trial EEG recordings collected during silent reading. To evaluate whether the system extracts meaningful information from these EEG signals, the results are compared with a random baseline. In nine subjects, the proposed pipeline outperforms the random baseline, achieving a mean cosine similarity of 0.181 +- 0.022 compared to 0.139 +- 0.029 for the baseline, corresponding to a relative improvement of 30.45%. Statistical analysis further confirms that this improvement is significant, following a strict evaluation workflow where inference is performed without access to ground-truth labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。