针对手语识别数据少的问题,提出轻量级模型提升识别准确率。
TransSLR: A Lightweight Transformer for Sign Language Recognition

- 用关键点序列训练轻量时序变换器,不依赖视觉外观。
- 在CASL-W60上达到80.39%准确率,提升10.46个百分点。
- 模型小、速度快,适合低资源设备部署。
针对未充分代表的手语(如中非手语,CASL)自动识别难题,现有基准CASL-W60的最佳准确率为69.93%,而利用高资源模型微调的通用方法无法缩小差距。这主要源于数据规模有限以及CASL与WLASL等大规模语料间存在显著词汇与视觉域差异,导致预训练表征基本无效。为此,我们提出TransSLR——一个从零开始训练的轻量级时序变换器编码器,输入为64帧归一化姿态关键点序列,通过平均池化与分类头实现识别。该模型基于几何关键点而非原始RGB图像,实现无需依赖书写者外观的泛化能力。在CASL-W60基准上,TransSLR取得80.39%的新最佳准确率,相较此前最优提升10.46%。此外,其仅编码器结构大幅降低计算开销,利于资源受限环境部署。我们在该基准上对比了基于RGB和多模态的基线模型,验证了TransSLR的卓越性能。
原文摘要 · Abstract (English)
Automated Sign Language Recognition for under-represented languages remains a largely unsolved problem. Central African Sign Language (CASL) exemplifies this gap: the only available bench-mark, CASL-W60, has a best reported accuracy of 69.93%, and we show that the common heuristic of fine-tuning high-resource models fails to close it. This failure stems from two compounding factors: the limited scale of available CASL data and the significant lexical and visual domain gap between CASL and large-scale corpora such as WLASL, which renders pre-trained representations largely uninformative. To address this, we propose TransSLR, a lightweight Temporal Transformer Encoder trained from scratch on 64-frame normalized pose sequences, with average pooling and a classification head. By operating on geometric keypoint representations rather than raw RGB, TransSLR achieves signer-independent generalization without relying on visual appearance. On the CASL-W60 benchmark, TransSLR establishes a new state-of-the-art accuracy of 80.39%, surpassing the prior best by +10.46%. Beyond accuracy, our encoder-only design significantly reduces computational overhead, making deployment feasible in resource-constrained environments. We conduct extensive experiments on the CASL-W60 benchmark, comparing against RGB-based and multimodal baselines, and demonstrate that TransSLR achieves state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。