用大模型解码脑电语音,跨数据集训练实现前所未有的泛化能力。
BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding
- 融合预训练语音模型与脑电数据,通过混合损失优化音素与词级预测。
- 跨数据集训练无需微调即提升性能,验证了卓越的泛化能力。
- 支持高精度音素路径和毫秒级快速生成,适合实时脑机接口应用。
从皮层内记录中解码连续语音是脑机接口的核心挑战,具有改变失语症患者沟通方式的潜力。尽管近期微电极阵列(MEA)解码器表现优异,但其性能受限于现有数据集规模小、对会话间差异敏感,且跨被试泛化能力尚未验证。我们提出BrainWhisperer,一种将高分辨率MEA记录与大规模预训练自动语音识别(ASR)模型结合的神经语音解码器。基于可解释性研究发现Whisper编码器能学习音素选择性表征并具有局部注意力特性,我们训练了一个定制版Whisper,修改为处理神经特征,并采用混合目标函数:在第三编码层预测音素使用CTC损失,词级预测使用交叉熵损失。引入领域知识改进包括窗口自注意力以捕捉发音连续性、分月/日的层次化低秩投影以应对非平稳性,以及个体特异性嵌入器实现跨被试训练。在公开可用的MEA数据集(Card et al.)上评估,BrainWhisperer达到或超越现有最优解码器性能。关键突破在于跨数据集训练即使在单个数据集上也无需微调即可提升表现,展现出前所未有的泛化能力。该模型支持双路径解码:高精度音素路径配合外部语言模型重打分,以及高速直接文本生成路径,实现低于100ms推理且硬件需求极低。
原文摘要 · Abstract (English)
Decoding continuous speech from intracortical recordings is a central challenge for brain-computer interfaces (BCIs), with transformative potential for individuals with conditions that impair their ability to speak. While recent microelectrode array (MEA) decoders achieve impressive accuracy, their performance is fundamentally limited by the small size of existing datasets, they remain brittle to session-to-session variability, and their ability to generalize across participants remains unexplored. We introduce BrainWhisperer, a neural speech decoder that integrates high-resolution MEA recordings with a large pretrained automatic speech recognition (ASR) model. Building on interpretability findings showing that Whisper's encoder learns phoneme-selective representations with localized attention, we train a customized version of Whisper, modified to process neural features, using a hybrid objective that combines CTC loss on phonemes--predicted from the third encoder layer--and cross-entropy loss on word tokens. We introduce domain-informed modifications including windowed self-attention to capture articulatory continuity, hierarchical month/day-specific low-rank projections to address non-stationarity, and subject-specific embedders enabling cross-subject training. Evaluated on a publicly available MEA dataset (Card et al.), BrainWhisperer matches or outperforms prior state-of-the-art decoders. Critically, cross-dataset training improves performance even on individual datasets without fine-tuning, demonstrating unprecedented generalization. The model supports dual decoding paths: a high-accuracy phoneme-based path with external language model rescoring, and a fast direct text generation path enabling sub-100ms inference with minimal hardware requirements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。