arXiv:2503.16855cs.CV2025-03被引 1

用堆叠Transformer捕捉手语动态,无需预设骨骼图结构。

Stack Transformer Based Spatial-Temporal Attention Model for Dynamic Sign Language and Fingerspelling Recognition

  • 分层堆叠Transformer,分别处理帧内关节关系与跨帧时序依赖。
  • 在JSL和KSL数据集上达到手指拼写识别新SOTA,WLASL上超越纯骨架方法。
  • 从零训练即达顶尖性能,适合低资源手语识别研究者使用。

基于手势的手语识别(SLR)是聋人与非聋人之间的重要沟通桥梁。尽管图卷积网络(GCNs)被广泛使用,但其依赖固定骨骼图结构存在局限。为此,我们提出一种新型Transformer架构——序列时空注意力网络(SSTAN)。该模型采用分层堆叠设计,依次引入空间多头注意力(MHA)捕捉帧内关节间关系,以及时间多头注意力(MHA)建模跨帧长程依赖。该方法无需预定义图结构,可高效学习复杂时空模式。我们在多个大规模数据集(WLASL、JSL、KSL)上进行了充分实验。关键发现为:模型完全从零训练,即可在挑战性手指拼写类别(JSL和KSL)中达到当前最优(SOTA)表现;同时,在仅使用骨架信息的WLASL上也建立新SOTA,优于多种依赖复杂自监督预训练的方法。结果表明该模型具备高数据效率,能有效捕捉手语的精细动态特征。官方代码已开源:https://github.com/K-Hirooka-Aizu/skeleton-slr-transformer。

原文摘要 · Abstract (English)

Hand gesture-based Sign Language Recognition (SLR) serves as a crucial communication bridge between deaf and non-deaf individuals. While Graph Convolutional Networks (GCNs) are common, they are limited by their reliance on fixed skeletal graphs. To overcome this, we propose the Sequential Spatio-Temporal Attention Network (SSTAN), a novel Transformer-based architecture. Our model employs a hierarchical, stacked design that sequentially integrates Spatial Multi-Head Attention (MHA) to capture intra-frame joint relationships and Temporal MHA to model long-range inter-frame dependencies. This approach allows the model to efficiently learn complex spatio-temporal patterns without predefined graph structures. We validated our model through extensive experiments on diverse, large-scale datasets (WLASL, JSL, and KSL). A key finding is that our model, trained entirely from scratch, achieves state-of-the-art (SOTA) performance in the challenging fingerspelling categories (JSL and KSL). Furthermore, it establishes a new SOTA for skeleton-only methods on WLASL, outperforming several approaches that rely on complex self-supervised pre-training. These results demonstrate our model's high data efficiency and its effectiveness in capturing the intricate dynamics of sign language. The official implementation is available at our GitHub repository: \href{https://github.com/K-Hirooka-Aizu/skeleton-slr-transformer}{https://github.com/K-Hirooka-Aizu/skeleton-slr-transformer}.

手语识别Transformer时空建模骨架数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。