用脑电+眼动数据实现开放词汇文本生成与情感分类
ETS: Open Vocabulary Electroencephalography-To-Text Decoding and Sentiment Classification
- 融合脑电与眼动数据，实现开放词汇的文本还原
- 文本生成效果优于基线，情感分类F1提升10%
- 跨被试、跨数据源表现稳定，适合实际应用
利用非侵入式脑电图（EEG）解码自然语言仍是神经科学与机器学习中的重大挑战，尤其在开放词汇场景下，传统方法受噪声和个体差异影响大。以往研究在小规模封闭词汇上取得高精度，但在开放词汇上仍表现不佳。本文提出ETS框架，结合同步眼动追踪数据，完成两项任务：(1) 开放词汇文本生成；(2) 被感知语言的情感分类。模型在EEG到文本解码中达到更高BLEU与Rouge分数，在基于EEG的三元情感分类中F1最高提升10%，显著优于监督基线。此外，模型在不同被试和数据源上均表现良好，展现出构建高性能开放词汇脑电转文本系统的重要潜力。
原文摘要 · Abstract (English)
Decoding natural language from brain activity using non-invasive electroencephalography (EEG) remains a significant challenge in neuroscience and machine learning, particularly for open-vocabulary scenarios where traditional methods struggle with noise and variability. Previous studies have achieved high accuracy on small-closed vocabularies, but it still struggles on open vocabularies. In this study, we propose ETS, a framework that integrates EEG with synchronized eye-tracking data to address two critical tasks: (1) open-vocabulary text generation and (2) sentiment classification of perceived language. Our model achieves a superior performance on BLEU and Rouge score for EEG-To-Text decoding and up to 10% F1 score on EEG-based ternary sentiment classification, which significantly outperforms supervised baselines. Furthermore, we show that our proposed model can handle data from various subjects and sources, showing great potential for high performance open vocabulary eeg-to-text system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。