用脑电和肌电信号重建未见过的句子,迈向开放词汇神经通信
Reconstructing Unseen Sentences from Speech-related Biosignals for Open-vocabulary Neural Communication
- 基于高密度脑电与肌电的音素级信息,重建未知语句
- 在多种语音模式下实现未见过句子的合成,验证可行性
- 为患者个性化康复提供神经通信新思路
脑-语音(BTS)系统通过将神经活动直接转换为语言表达,革新了人类沟通方式。尽管近期非侵入式BTS研究主要聚焦于预定义词句的解码,但要实现接近自然对话的开放词汇神经通信,必须能够解码不受限的口语。同时,有效整合来自语音的多种信号对发展个性化、自适应的神经通信与康复解决方案至关重要。本研究探索了利用高密度脑电图(EEG)信号中提取的音素级信息,独立或结合肌电(EMG)信号,重构不同语音模式下的未见句子的潜力。此外,我们分析了影响音素解码准确性的因素,并提供了神经生理学见解,以进一步提升基于EEG的解码性能。研究结果证实,基于生物信号的句子级语音合成可成功重建未见过的句子,标志着向适配多样化患者需求与状况的开放词汇神经通信系统迈出关键一步。本研究还为基于EEG解码技术的通信与康复解决方案开发提供了重要启示。
原文摘要 · Abstract (English)
Brain-to-speech (BTS) systems represent a groundbreaking approach to human communication by enabling the direct transformation of neural activity into linguistic expressions. While recent non-invasive BTS studies have largely focused on decoding predefined words or sentences, achieving open-vocabulary neural communication comparable to natural human interaction requires decoding unconstrained speech. Additionally, effectively integrating diverse signals derived from speech is crucial for developing personalized and adaptive neural communication and rehabilitation solutions for patients. This study investigates the potential of speech synthesis for previously unseen sentences across various speech modes by leveraging phoneme-level information extracted from high-density electroencephalography (EEG) signals, both independently and in conjunction with electromyography (EMG) signals. Furthermore, we examine the properties affecting phoneme decoding accuracy during sentence reconstruction and offer neurophysiological insights to further enhance EEG decoding for more effective neural communication solutions. Our findings underscore the feasibility of biosignal-based sentence-level speech synthesis for reconstructing unseen sentences, highlighting a significant step toward developing open-vocabulary neural communication systems adapted to diverse patient needs and conditions. Additionally, this study provides meaningful insights into the development of communication and rehabilitation solutions utilizing EEG-based decoding technologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。