用文本描述对齐脑电图，实现零样本解码新方法
EEG-CLIP : Learning EEG representations from natural language descriptions
- 通过对比学习将脑电图与临床文本映射到共享向量空间
- 在少样本和零样本设置下实现多种脑电解码任务
- 适合想快速验证新解码任务的研究者使用
针对脑电图（EEG）解码模型通常仅适用于单一任务的问题，本文提出一种任务无关的通用方法：训练深度网络将临床脑电记录与其对应的文本报告相互匹配。受计算机视觉中图像-文本匹配启发，我们构建了EEG-CLIP框架，通过对比学习使脑电时间序列与对应临床文本描述在共享嵌入空间中对齐。我们在多种少样本和零样本设置下评估其性能，结果表明EEG-CLIP能非平凡地对齐文本与脑电表示。该方法为学习通用脑电表征提供了新路径，可支持零样本解码或以更少标注数据训练特定任务模型。代码已开源：https://github.com/tidiane-camaret/EEGClip
原文摘要 · Abstract (English)
Deep networks for electroencephalogram (EEG) decoding are often only trained to solve one specific task, such as pathology or age decoding. A more general task-agnostic approach is to train deep networks to match a (clinical) EEG recording to its corresponding textual medical report and vice versa. This approach was pioneered in the computer vision domain matching images and their text captions and subsequently allowed to do successful zero-shot decoding using textual class prompts. In this work, we follow this approach and develop a contrastive learning framework, EEG-CLIP, that aligns the EEG time series and the descriptions of the corresponding clinical text in a shared embedding space. We investigated its potential for versatile EEG decoding, evaluating performance in a range of few-shot and zero-shot settings. Overall, we show that EEG-CLIP manages to non-trivially align text and EEG representations. Our work presents a promising approach to learn general EEG representations, which could enable easier analyses of diverse decoding questions through zero-shot decoding or training task-specific models from fewer training examples. The code for reproducing our results is available at https://github.com/tidiane-camaret/EEGClip
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。