用语义对齐提升脑电转文字准确率，让大脑想法更精准被读取。
SEE: Semantically Aligned EEG-to-Text Translation
- 将跨模态码本与语义匹配模块嵌入BART模型，缓解脑电与文本的领域差异。
- 在ZuCo数据集上实现更优的文本生成效果，显著提升解码准确性。
- 适合脑机接口、神经语言解码研究者，尤其关注低资源场景应用。
将神经生理信号解码为语言是脑机接口（BCI）中的重要研究方向。脑电图（EEG）因其非侵入性、易用性和低成本成为该领域的常用方法。然而，现有脑电到文本的解码方法受限于脑电与原始文本间巨大的领域差距、固有的数据偏差以及小规模封闭词汇表。本文提出SEE：语义对齐的脑电到文本翻译，通过将两个模块无缝集成至预训练的BART语言模型中，以提升解码性能。这两个模块包括：(1) 跨模态码本，用于学习跨模态表示以增强特征一致性并缓解领域差距；(2) 语义匹配模块，充分利用预训练文本表示，对齐从脑电-文本配对中提取的多模态特征，同时考虑因误负样本（即不同脑电-文本对但语义相似）带来的噪声影响。在苏黎世认知语言处理语料库（ZuCo）上的实验结果表明，SEE方法有效提升了脑电到文本解码的可行性。
原文摘要 · Abstract (English)
Decoding neurophysiological signals into language is of great research interest within brain-computer interface (BCI) applications. Electroencephalography (EEG), known for its non-invasiveness, ease of use, and cost-effectiveness, has been a popular method in this field. However, current EEG-to-Text decoding approaches face challenges due to the huge domain gap between EEG recordings and raw texts, inherent data bias, and small closed vocabularies. In this paper, we propose SEE: Semantically Aligned EEG-to-Text Translation, a novel method aimed at improving EEG-to-Text decoding by seamlessly integrating two modules into a pre-trained BART language model. These two modules include (1) a Cross-Modal Codebook that learns cross-modal representations to enhance feature consolidation and mitigate domain gap, and (2) a Semantic Matching Module that fully utilizes pre-trained text representations to align multi-modal features extracted from EEG-Text pairs while considering noise caused by false negatives, i.e., data from different EEG-Text pairs that have similar semantic meanings. Experimental results on the Zurich Cognitive Language Processing Corpus (ZuCo) demonstrate the effectiveness of SEE, which enhances the feasibility of accurate EEG-to-Text decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。