用3D CNN和LSTM实时识别手语，助力聋哑人沟通
Real-Time American Sign Language Recognition Using 3D Convolutional Neural Networks and LSTM: Architecture, Training, and Deployment
- 结合3D卷积与LSTM，捕捉手势时空特征
- 在多个数据集上达0.71至0.99的F1分数
- 可部署于云端及OAK-D边缘设备，支持实时推理
本文提出一种基于3D卷积神经网络（3D CNN）与长短期记忆（LSTM）网络的混合深度学习架构，实现对美国手语（ASL）的实时识别。系统通过摄像头视频流识别词级手语动作，旨在帮助全球超过7000万聋哑人士跨越交流障碍。模型利用3D卷积提取视频帧的时空特征，再通过LSTM层建模手语动作的时序依赖关系。在WLASL数据集（2000个常用词）、ASL-LEX词汇库（约2700个手语符号）以及100个专家标注的手语样本上训练，各类别F1分数在0.71至0.99之间。模型部署于AWS云平台，并具备在OAK-D摄像头上的边缘推理能力。文章讨论了架构设计、训练方法、评估指标及实际应用中的部署考量。
原文摘要 · Abstract (English)
This paper presents a real-time American Sign Language (ASL) recognition system utilizing a hybrid deep learning architecture combining 3D Convolutional Neural Networks (3D CNN) with Long Short-Term Memory (LSTM) networks. The system processes webcam video streams to recognize word-level ASL signs, addressing communication barriers for over 70 million deaf and hard-of-hearing individuals worldwide. Our architecture leverages 3D convolutions to capture spatial-temporal features from video frames, followed by LSTM layers that model sequential dependencies inherent in sign language gestures. Trained on the WLASL dataset (2,000 common words), ASL-LEX lexical database (~2,700 signs), and a curated set of 100 expert-annotated ASL signs, the system achieves F1-scores ranging from 0.71 to 0.99 across sign classes. The model is deployed on AWS infrastructure with edge deployment capability on OAK-D cameras for real-time inference. We discuss the architecture design, training methodology, evaluation metrics, and deployment considerations for practical accessibility applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。