用大模型理解无声肌电信号，实现高精度文字转换。
Can LLMs Understand Unvoiced Speech? Exploring EMG-to-Text Conversion with LLMs
- 设计新适配模块，将肌电特征映射到大模型输入空间
- 仅需6分钟数据，字错误率低至0.49，性能提升近20%
- 为无法发声者提供新沟通方式，适合残障辅助领域
无声肌电（unvoiced EMG）是无法发声者的重要沟通工具。以往方法多依赖有声与无声肌电信号配对及语音数据，不适用于该群体。随着大语言模型（LLMs）在语音识别中的应用，我们探索其理解无声语音的潜力。为此，提出一种仅基于无声肌电学习的新方法，设计了肌电适配模块，将肌电特征映射至大模型输入空间，在闭词汇无声肌电转文字任务中实现平均词错误率（WER）0.49。即使数据量保守仅为六分钟，性能仍比专用模型高出近20%。尽管大模型已可扩展至音频等新模态，但理解如无声肌电这类发音生物信号仍具挑战。本工作迈出关键第一步，使大模型能通过表面肌电理解无声语音。
原文摘要 · Abstract (English)
Unvoiced electromyography (EMG) is an effective communication tool for individuals unable to produce vocal speech. However, most prior methods rely on paired voiced and unvoiced EMG signals, along with speech data, for EMG-to-text conversion, which is not practical for such individuals. Given the rise of large language models (LLMs) in speech recognition, we explore their potential to understand unvoiced speech. To this end, we address the challenge of learning from unvoiced EMG alone and propose a novel EMG adaptor module that maps EMG features into an LLM's input space, achieving an average word error rate (WER) of 0.49 on a closed-vocabulary unvoiced EMG-to-text task. Even with a conservative data availability of just six minutes, our approach improves performance over specialized models by nearly 20%. While LLMs have been shown to be extendable to new language modalities -- such as audio -- understanding articulatory biosignals like unvoiced EMG remains more challenging. This work takes a crucial first step toward enabling LLMs to comprehend unvoiced speech using surface EMG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。