arXiv:2409.07426cs.CV2024-09被引 8

用深度神经网络与可解释性技术,实现高精度手语识别。

Deep Neural Network-Based Sign Language Recognition: A Comprehensive Approach Using Transfer Learning with Explainability

  • 结合迁移学习与预处理优化,提升手语图像分类性能。
  • 在不丹手语数据集上达到98.90%准确率,表现优异。
  • 通过SHAP方法揭示模型决策依据,适合无障碍应用开发。

为促进残障人士的包容性并确保有效沟通,手语识别(SLR)至关重要。该研究提出一种基于深度神经网络的全自动手语识别方案,融合了先进的预处理方法以优化整体性能。采用ResNet、Inception、Xception和VGG等架构对手语图像进行分类,并构建了DNN架构与预处理模块集成。后处理阶段使用基于合作博弈论的SHAP深度解释器,量化特定特征对模型输出的影响。实验基于不丹手语数据集(BSL),结果显示,结合DNN的ResNet50模型在训练中达到98.90%的准确率。通过SHAP方法验证了模型的可解释性与信息清晰度。由于其显著的鲁棒性和可靠性,该方法可应用于全自动手语识别系统开发。

原文摘要 · Abstract (English)

To promote inclusion and ensuring effective communication for those who rely on sign language as their main form of communication, sign language recognition (SLR) is crucial. Sign language recognition (SLR) seamlessly incorporates with diverse technology, enhancing accessibility for the deaf community by facilitating their use of digital platforms, video calls, and communication devices. To effectively solve this problem, we suggest a novel solution that uses a deep neural network to fully automate sign language recognition. This methodology integrates sophisticated preprocessing methodologies to optimise the overall performance. The architectures resnet, inception, xception, and vgg are utilised to selectively categorise images of sign language. We prepared a DNN architecture and merged it with the pre-processing architectures. In the post-processing phase, we utilised the SHAP deep explainer, which is based on cooperative game theory, to quantify the influence of specific features on the output of a machine learning model. Bhutanese-Sign-Language (BSL) dataset was used for training and testing the suggested technique. While training on Bhutanese-Sign-Language (BSL) dataset, overall ResNet50 with the DNN model performed better accuracy which is 98.90%. Our model's ability to provide informational clarity was assessed using the SHAP (SHapley Additive exPlanations) method. In part to its considerable robustness and reliability, the proposed methodological approach can be used to develop a fully automated system for sign language recognition.

手语识别深度学习可解释性迁移学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。