arXiv:2601.17041cs.CVcs.AI2026-01

融合手势与视觉数据,提升阿拉伯手语识别准确率。

Arabic Sign Language Recognition using Multimodal Approach

  • 用深度网络处理手势数据,图像网络微调VGG16增强特征提取。
  • 在自建数据集上实现13个词正确识别,总体准确率达78%。
  • 适合研究多模态融合与残障人士交互系统的开发者参考。

阿拉伯手语(ArSL)是聋哑人群体的重要沟通方式。现有识别系统多依赖单一传感器如Leap Motion或RGB摄像头,难以精准捕捉复杂手部姿态和三维运动。本文提出一种融合Leap Motion与RGB摄像头数据的多模态方法,设计双分支网络:一个定制的密集神经网络处理手势数据,采用丢弃层与L2正则化;另一个基于微调VGG16的图像子网络结合数据增强。两路特征在融合模型中拼接,经全连接层后通过SoftMax分类,分析手势的空间与时间特征。在包含18个ArSL词汇的自建数据集上测试,13个词汇被正确识别,整体准确率为78%。结果表明多模态融合具备可行性,也为后续优化与数据扩展提供依据。

原文摘要 · Abstract (English)

Arabic Sign Language (ArSL) is an essential communication method for individuals in the Deaf and Hard-of-Hearing community. However, existing recognition systems face significant challenges due to their reliance on single sensor approaches like Leap Motion or RGB cameras. These systems struggle with limitations such as inadequate tracking of complex hand orientations and imprecise recognition of 3D hand movements. This research paper aims to investigate the potential of a multimodal approach that combines Leap Motion and RGB camera data to explore the feasibility of recognition of ArSL. The system architecture includes two parallel subnetworks: a custom dense neural network for Leap Motion data, incorporating dropout and L2 regularization, and an image subnetwork based on a fine-tuned VGG16 model enhanced with data augmentation techniques. Feature representations from both modalities are concatenated in a fusion model and passed through fully connected layers, with final classification performed via SoftMax activation to analyze spatial and temporal features of hand gestures. The system was evaluated on a custom dataset comprising 18 ArSL words, of which 13 were correctly recognized, yielding an overall accuracy of 78%. These results offer preliminary insights into the viability of multimodal fusion for sign language recognition and highlight areas for further optimization and dataset expansion.

手语识别多模态深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。