从脑电基础模型中提炼分层关键知识,让小模型更轻更快。
DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models

- 按输入动态选择教师模型的中间层进行知识蒸馏
- 蒸馏后小模型参数减少90%以上,推理速度提升3倍
- 适合资源受限场景下的脑电信号实时分析
脑电基础模型(EFM)通过大规模预训练和微调,在跨被试、跨任务上表现优异。我们发现:(i) 任务适配后的EFM虽解码性能强,但作为推理主干存在巨大开销,知识蒸馏是优化紧凑学生模型的自然路径;(ii) 直接从固定教师表示蒸馏会低估知识,因任务判别信息分散在中间层而非最终层。为此提出DLink(分层与主导知识蒸馏),一个基于谱引导的蒸馏框架,采用输入感知层路由机制,将教师模型各层知识按需聚合。利用轻量路由器对每条输入动态整合教师层,并通过幅值与相位谱对齐缓解压缩带来的谱失真。学生模型先投影再压缩,教师与路由器仅用于训练。在四个脑电基准测试上,DLink显著提升紧凑学生的性能,优于轻量基线,缩小与微调EFM的差距,同时大幅降低参数量、浮点运算量及仅用CPU时的推理延迟。
原文摘要 · Abstract (English)
EEG foundation models (EFMs) achieve strong cross-subject and cross-task generalization through large-scale pretraining and downstream fine-tuning. Through empirical analysis, we observe that (i) task-adapted EFMs provide strong decoding performance but incur substantial overhead when retained as inference backbones, making knowledge distillation a natural route for optimizing compact students; and (ii) direct distillation from a fixed teacher representation underutilizes EFM knowledge, as task-discriminative information is distributed across intermediate layers rather than concentrated in the final layer. These observations motivate DLink (Distilling Layer-wise and Dominant Knowledge), a spectrally guided distillation framework with input-conditioned layer routing for transferring EFM knowledge into compact students. DLink uses a lightweight router to aggregate teacher layers for each input, and aligns magnitude and phase spectra to mitigate compression-induced spectral distortion in learned representations. The routed teacher knowledge is internalized by a project-then-compress student; the teacher and router are used only during training. Experiments on four EEG benchmarks show that DLink improves matched compact students and remains competitive with lightweight baselines, narrowing the gap to fine-tuned EFMs while substantially reducing parameters, FLOPs, and CPU-only inference latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。