用视觉和时序模型做老师,蒸馏训练脑电基础模型,数据少一半效果还更好。
Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation
- 用多个现成大模型作教师,通过可学习门控融合特征进行两阶段蒸馏。
- 仅用25%数据就超越自监督方法,在9个任务12个数据集上表现更优。
- 适合脑电研究者、医疗AI开发者,尤其数据稀缺场景下使用。
脑电图(EEG)基础模型预训练长期依赖自监督掩码重建,这一范式源自视觉与语言模型的成功,但EEG数据采集成本高、信噪比低,导致难以规模化并捕捉深层神经语义。本文提出:能否借助已有模态的成熟基础模型来启动EEG模型预训练?我们验证了主流视觉与时间序列模型在EEG领域具有惊人迁移能力。为此,提出多教师蒸馏预训练(MTDP)框架,分两阶段进行:第一阶段引入可学习门控网络,通过掩码潜在表示去噪目标融合多个教师(如DINOv3和Chronos)的特征;第二阶段将融合后的表示蒸馏至EEG基础模型。在9个下游任务和12个数据集上的大量评估表明,基于MTDP的模型优于自监督基线,且仅需25%预训练数据。
原文摘要 · Abstract (English)
Pretraining for electroencephalogram (EEG) foundation models has predominantly relied on self-supervised masked reconstruction, a paradigm largely adapted from and inspired by the success of vision and language foundation models. However, unlike images and text, EEG datasets are notoriously expensive to collect and characterized by low signal-to-noise ratio. These challenges introduce difficulties in scaling the EEG foundation models and capturing the underlying neural semantics through reconstruction. In this work, we ask the question: can we stand on the shoulders of well-established foundation models from well-represented modalities to bootstrap the pretraining of EEG foundation models? We first demonstrate that mainstream foundation models, such as those from vision and time series, transfer surprisingly well to EEG domain. To this end, we propose the Multi-Teacher Distillation Pretraining (MTDP) framework for pretraining EEG foundation models via a two-stage multi-teacher distillation. In the first stage, we introduce a learnable gating network to fuse representations from diverse teachers (e.g., DINOv3 and Chronos) via a masked latent denoising objective. In the second stage, we distill the fused representation into an EEG foundation model. Extensive evaluations across 9 downstream tasks and 12 datasets demonstrate that our MTDP-based EEG foundation model outperforms its self-supervised counterparts while requiring only 25% of the pretraining data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。