用骨骼点识别土耳其手语，轻量高效可实时运行
TSLFormer: A Lightweight Transformer Model for Turkish Sign Language Recognition Using Skeletal Landmarks
- 以骨骼关键点为输入，用Transformer建模手语序列
- 在36,000+样本上达到媲美主流模型的准确率
- 适合移动端和助听障碍者实时沟通系统
本研究提出TSLFormer，一种轻量且鲁棒的词级土耳其手语（TSL）识别模型，将手语动作视为有序的、类字符串语言。不使用原始RGB或深度视频，仅采用谷歌Mediapipe提取的手部与躯干骨骼关键点作为输入，实现高效降维并保留关键语义信息。方法借鉴自然语言处理中Transformer的成功经验，将手语识别视为序列到序列翻译任务。由于TSLFormer使用自注意力机制，能有效捕捉手势序列中的时间共现关系，突出随动作展开的有意义运动模式。在包含超过36,000个样本和227个不同词汇的AUTSL数据集上评估，TSLFormer在极低计算成本下实现具有竞争力的性能。结果表明，基于关节点的输入足以支持面向听障人士的实时、移动化辅助通信系统。
原文摘要 · Abstract (English)
This study presents TSLFormer, a light and robust word-level Turkish Sign Language (TSL) recognition model that treats sign gestures as ordered, string-like language. Instead of using raw RGB or depth videos, our method only works with 3D joint positions - articulation points - extracted using Google's Mediapipe library, which focuses on the hand and torso skeletal locations. This creates efficient input dimensionality reduction while preserving important semantic gesture information. Our approach revisits sign language recognition as sequence-to-sequence translation, inspired by the linguistic nature of sign languages and the success of transformers in natural language processing. Since TSLFormer uses the self-attention mechanism, it effectively captures temporal co-occurrence within gesture sequences and highlights meaningful motion patterns as words unfold. Evaluated on the AUTSL dataset with over 36,000 samples and 227 different words, TSLFormer achieves competitive performance with minimal computational cost. These results show that joint-based input is sufficient for enabling real-time, mobile, and assistive communication systems for hearing-impaired individuals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。