对比了脑电数据分词策略,发现简单方法也能有效用于大模型训练。
A Systematic Evaluation of Sample-Level Tokenization Strategies for MEG Foundation Models
- 提出基于自编码器的可学习分词方法,对比固定分词策略
- 两种方法在重建精度和下游任务表现上基本相当
- 适合开发脑电领域基础模型的研究者参考
自然语言处理的成功推动了神经影像领域大规模基础模型的发展。这类模型通常需要将连续的神经时间序列数据离散化,称为‘分词’。然而,不同分词策略对神经数据的影响尚不明确。本文系统评估了应用于脑磁图(MEG)数据的Transformer架构大神经影像模型(LNMs)的样本级分词策略。通过比较可学习与不可学习分词器,在信号重建保真度、后续基础建模性能(分词预测、生成数据生物合理性、个体信息保留、下游任务表现)等方面进行评估。针对可学习分词器,提出一种基于自编码器的新方法。实验在三个公开MEG数据集上展开,涵盖不同采集站点、扫描仪和实验范式。结果表明,可学习与不可学习的离散化方案均实现高重建准确率,且在多数评估指标上表现相近,说明简单的固定样本级分词策略可用于神经基础模型的开发。代码已开源:https://github.com/OHBA-analysis/Cho2026_Tokenizer。
原文摘要 · Abstract (English)
Recent success in natural language processing has motivated growing interest in large-scale foundation models for neuroimaging data. Such models often require discretization of continuous neural time series data, a process referred to as 'tokenization'. However, the impact of different tokenization strategies for neural data is currently poorly understood. In this work, we present a systematic evaluation of sample-level tokenization strategies for transformer-based large neuroimaging models (LNMs) applied to magnetoencephalography (MEG) data. We compare learnable and non-learnable tokenizers by examining their signal reconstruction fidelity and their impact on subsequent foundation modeling performance (token prediction, biological plausibility of generated data, preservation of subject-specific information, and performance on downstream tasks). For the learnable tokenizer, we introduce a novel approach based on an autoencoder. Experiments were conducted on three publicly available MEG datasets spanning different acquisition sites, scanners, and experimental paradigms. Our results show that both learnable and non-learnable discretization schemes achieve high reconstruction accuracy and broadly comparable performance across most evaluation criteria, suggesting that simple fixed sample-level tokenization strategies can be used in the development of neural foundation models. The code is available at https://github.com/OHBA-analysis/Cho2026_Tokenizer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。