解决脑电图持续监测中的内存瓶颈,实现高效长时分析
S-CEReBrO: Breaking the Memory Barrier in Continuous EEG Monitoring

- 用分窗交替注意力机制替代全局注意力,内存不随信号变长而增长
- 可处理100倍于传统自注意力的长信号,内存仅需低秩线性注意力的55%
- 适用于长期脑电监测场景,尤其适合资源受限设备部署
基础模型为脑电图(EEG)分析提供了新范式,利用海量无标签数据学习通用表征。然而,基于Transformer的架构存在关键瓶颈:全局注意力机制使记忆状态与信号长度耦合,导致连续监测时内存溢出。为此,我们提出S-CEReBrO(Streaming CEReBrO),一种专为连续监测设计的CEReBrO架构演进版本。其创新的分窗交替注意力机制将注意力计算分解为固定大小的时空窗口,确保仅活跃窗口需驻留注意力映射,从而维持恒定的键值缓存内存。实证扩展分析表明,该机制可处理比全自注意力长100倍、比低秩线性注意力长3倍的信号;相较于低秩线性注意力,在长上下文上内存减少55%,推理吞吐量提升2.1倍。模型在超过25,000小时、来自逾12,000名受试者的数据上预训练,于11项下游任务中达到7项最优表现,参数量最多减少60%。本工作推动了高效、通用且连续的脑电监测技术的发展。配套代码库已公开。
原文摘要 · Abstract (English)
Foundation models offer a promising paradigm for Electroencephalography (EEG) analysis, leveraging generalizable representations from vast unlabeled datasets. Yet, Transformer-based architectures face a critical bottleneck: global attention mechanisms couple the attention memory state to the signal duration, causing memory overflow during continuous monitoring. To address this, we introduce S-CEReBrO (Streaming CEReBrO), an evolution of the CEReBrO architecture designed for continuous monitoring. Our novel Windowed Alternating Attention mechanism factorizes attention computation into fixed-size spatiotemporal windows, guaranteeing constant KV cache memory as only the active window requires resident attention maps. Empirical scaling analysis confirms that windowed alternating attention can process signals 100X longer than full self-attention and 3X longer than low-rank linear attention. Compared to low-rank linear attention on long contexts, windowed alternating attention requires 55% of the memory while increasing inference throughput by 2.1X. Pre-trained on >25,000 hours of recordings from >12,000 subjects, S-CEReBrO achieves state-of-the-art performance on 7 of 11 downstream tasks, with up to 60% fewer parameters. This work represents a significant step toward the realization of efficient, generalizable, and continuous EEG monitoring. An accompanying code repository is available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。