统一建模手部与面部动作，提升连续手语识别准确率
USTM: Unified Spatial and Temporal Modeling for Continuous Sign Language Recognition
- 用Swin Transformer加轻量时序适配器联合建模空间与时间特征
- 在PHOENIX14等数据集上达到当前最优的单流RGB识别性能
- 无需多流输入或额外模态，适合实际部署的视觉手语系统
连续手语识别（CSLR）需要精确的时空建模以准确识别视频中的手势序列。现有方法多采用基于CNN的空间主干网络结合时序卷积或循环模块，难以捕捉精细的手部和面部线索，也无法有效建模长程时序依赖。为此，我们提出统一时空建模（USTM）框架，一个结合Swin Transformer主干与轻量级时序适配器及位置嵌入（TAPE）的时空编码器。该框架能同时捕获细粒度空间特征与短、长时程时间上下文，仅依赖RGB视频实现鲁棒手语识别，无需多流输入或辅助模态。在PHOENIX14、PHOENIX14T和CSL-Daily等基准数据集上的大量实验表明，USTM在基于RGB的方法中达到当前最优性能，且与多流方法相比仍具竞争力。结果验证了USTM在CSLR中的有效性与优势。代码已开源：https://github.com/gufranSabri/USTM。
原文摘要 · Abstract (English)
Continuous sign language recognition (CSLR) requires precise spatio-temporal modeling to accurately recognize sequences of gestures in videos. Existing frameworks often rely on CNN-based spatial backbones combined with temporal convolution or recurrent modules. These techniques fail in capturing fine-grained hand and facial cues and modeling long-range temporal dependencies. To address these limitations, we propose the Unified Spatio-Temporal Modeling (USTM) framework, a spatio-temporal encoder that effectively models complex patterns using a combination of a Swin Transformer backbone enhanced with lightweight temporal adapter with positional embeddings (TAPE). Our framework captures fine-grained spatial features alongside short and long-term temporal context, enabling robust sign language recognition from RGB videos without relying on multi-stream inputs or auxiliary modalities. Extensive experiments on benchmarked datasets including PHOENIX14, PHOENIX14T, and CSL-Daily demonstrate that USTM achieves state-of-the-art performance against RGB-based as well as multi-modal CSLR approaches, while maintaining competitive performance against multi-stream approaches. These results highlight the strength and efficacy of the USTM framework for CSLR. The code is available at https://github.com/gufranSabri/USTM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。