用视频识别手语并翻译成印地语等方言，助力聋哑人沟通。
Deep Learning-Based Sign Language Recognition from Videos and Cross-Lingual Translation to Indian Vernaculars
- 分两阶段：先用VideoMAE识出手语动作，再用NLLB-200翻译成方言
- 13类手语词识别准确率78%，支持印地语、泰卢固语、孟加拉语翻译
- 代码开源，适合残障技术、多语言翻译研究者参考
手语是全球聋哑人群体的主要交流方式，但能从视频中自动识别手语并翻译为自然语言的工具仍有限，尤其针对低资源印度语言。本文提出一个两阶段深度学习流程：(i) 使用微调的VideoMAE视频变换器对短手语视频片段进行分类，输出英语词汇标签；(ii) 利用Meta AI的NLLB-200多语言翻译模型将预测的英语标签翻译为印地语、泰卢固语和孟加拉语。分类模型在来自IIT Madras的AI4Bharat印度手语视频语料库的13类子集上微调，处理每段视频均匀采样的16帧图像，分辨率为224×224。在小规模学术设置下（13类，197个样本，80-20划分），模型训练准确率达99%，验证准确率为78%（15轮后）。通过混淆矩阵与分类报告分析了每类表现，识别出主要误判模式（如ugly、deaf、blind、hat、dress等形容词易混淆）。提供基于Streamlit的推理演示，用户上传视频后可返回对应的英语标签及三种方言翻译。讨论了当前局限（标签集小、孤立词而非连续手语、单人风格敏感性、单词翻译歧义）与未来方向（扩展至句级生成、更大词汇量）。代码已发布以支持复现。
原文摘要 · Abstract (English)
Sign language is a primary mode of communication for the global deaf and hard-of-hearing community, yet automated tools that recognize sign gestures from video and translate them into natural language text remain limited, particularly for low-resource Indian languages. We present a two-stage deep learning pipeline that (i) classifies short sign language video clips into English word labels using a fine-tuned VideoMAE video transformer, and (ii) translates the predicted English label into Hindi, Telugu, and Bengali using Meta AI's No Language Left Behind (NLLB-200) multilingual translation model. The classification model is fine-tuned on a 13-class subset of the AI4Bharat Indian Sign Language video corpus from IIT Madras, processing 16-frame clips sampled uniformly from each video at 224 x 224 resolution. Under a small-scale academic setting (13 classes, 197 clips, 80-20 split), the fine-tuned model reaches 99% training accuracy and 78% validation accuracy after 15 epochs. We provide a per-class breakdown via a confusion matrix and classification report, identify the dominant failure modes (confusable adjective pairs such as ugly, deaf, blind, hat, and dress), and describe a Streamlit-based inference demo that takes a user-uploaded video and returns the predicted English label alongside its Hindi, Telugu, and Bengali translations. We discuss the scope, limitations (small label set, isolated-word rather than continuous signing, single-signer style sensitivity, ambiguity of single-word machine translation), and directions for future work, including expanding to sentence-level generation and a larger vocabulary. Code is released to support reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。