arXiv:2607.22779cs.LGcs.AI2026-07

用多模态数据提升假肢手势识别准确率,解决传统方法难扩展的问题。

Multimodal Surface EMG Hand Gesture Recognition Using Query-Based Transformers for Prosthetic Control

论文配图:Multimodal Surface EMG Hand Gesture Recognition Using Query-Based Transformers for Prosthetic Control
图 1 · 摘自论文原文
  • 通过交叉注意力融合多模态信号,结合局部卷积与全局建模优势
  • 仅用肌电数据时平均准确率达72.33%~79.16%,加入惯性数据后提升至90%以上
  • 适合需要高精度、多手势控制的假肢系统研发人员参考

通过表面肌电(sEMG)进行手部手势识别是假肢控制的基础。尽管深度学习已成为该领域的主流方法,但现有架构在手势数量增加时性能下降明显,这源于解码复杂手势集的统计复杂性上升,以及依赖低延迟单模态卷积结构的局限性。卷积仅能捕捉局部特征,难以建模长序列模式;单模态设计也无法利用运动执行中的互补信息,如惯性数据和眼动追踪。为此,本文提出EMG-CrossFormer,一种端到端的混合卷积-变压器架构,可无缝整合任意数量的单模态编码器。该模型通过级联交叉注意力融合层整合各模态表示,并使用可学习的手势查询进行解码。在NinaPro四个数据集(DB2、DB3、DB7、DB10)上评估,仅使用sEMG时,平均准确率为72.33%、52.48%、79.16%和73.49%;加入惯性信号后,准确率提升至90.66%、80.40%、92.79%和92.06%。结果表明,联合局部-全局特征建模能有效提升sEMG解码性能,而多模态融合进一步显著增强效果,验证了两种设计思路在复杂手势识别中的价值。

原文摘要 · Abstract (English)

Hand gesture recognition via surface electromyography (sEMG) is fundamental to prosthetic control. In this field, deep learning approaches have become the gold standard. However, current architectures struggle to scale; model performance typically decreases as the number of hand movements increases. Performance degradation is tied to the increased statistical complexity of decoding expanded gesture sets and compounded by the limitations of state-of-the-art methods, which primarily rely on low-latency unimodal convolutional architectures. Convolutions operate locally, limiting model's ability to capture long-range sequential patterns. Unimodal setups cannot leverage complementary information from coordinated signals characterizing movement execution, such as inertial and eye-tracking data. These limitations motivate architectures that integrate local and global features across multimodal physiological sequences. To bridge this gap, this study introduces EMG-CrossFormer, an end-to-end hybrid convolutional-transformer for seamless multimodal integration. EMG-CrossFormer combines representations from an arbitrary number of unimodal encoders through cascaded cross-attention fusion layers, and decodes the fused representations using learnable gesture queries. EMG-CrossFormer was evaluated on four NinaPro datasets (DB2, DB3, DB7, and DB10) and benchmarked against six state-of-the-art models using an increasing number of modalities. Using only sEMG, EMG-CrossFormer achieved mean accuracies of 72.33%, 52.48%, 79.16%, and 73.49% on DB2, DB3, DB7, and DB10, respectively. Incorporating inertial signals improved performance to 90.66%, 80.40%, 92.79%, and 92.06%. These results show that joint local-global feature modeling improves sEMG-only decoding and that multimodal fusion substantially amplifies this benefit, underscoring the value of both design principles for complex hand gesture recognition.

假肢控制肌电识别多模态融合注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。