用图网络捕捉手语中的空间、时间与层次特征，提升识别效果。
Graph Network for Sign Language Tasks
- 构建混合图结构，分三模块捕捉手部区域间关系、帧间动态与多粒度特征。
- 在5个公开数据集上超越现有最优模型，无需额外标注信息。
- 提出文本驱动的伪标签预训练法，适用于无词义标注场景。
近年来手语研究受益于基于CNN的主干网络，这些网络主要来自传统计算机视觉任务（如物体识别、图像识别）。然而，这类方法擅长提取轮廓和纹理等特征，难以捕捉手语相关的特征。实际上，手语任务需关注手语相关区域，包括不同区域间的协作（如左右手区域）及单区域内有效内容。为捕捉此类区域相关特征，本文提出MixSignGraph，将手语序列表示为一组混合图，并设计三种图模块进行特征提取：局部手语图（LSG）模块学习单帧内跨区域特征关联，聚焦空间特征；时序手语图（TSG）模块追踪相邻帧间跨区域特征交互，聚焦时间特征；层级手语图（HSG）模块聚合同一区域在不同粒度特征图上的特征，聚焦层次特征。此外，为在无词义标注情况下进一步提升性能，提出一种简单但反直觉的文本驱动CTC预训练（TCP）方法，利用文本标签生成伪词义标签用于模型预训练。在当前五个公开手语数据集上的大量实验表明，所提模型表现优异，显著超越多个数据集上的现有最优模型，且不依赖任何额外线索。
原文摘要 · Abstract (English)
Recent advances in sign language research have benefited from CNN-based backbones, which are primarily transferred from traditional computer vision tasks (\eg object identification, image recognition). However, these CNN-based backbones usually excel at extracting features like contours and texture, but may struggle with capturing sign-related features. In fact, sign language tasks require focusing on sign-related regions, including the collaboration between different regions (\eg left hand region and right hand region) and the effective content in a single region. To capture such region-related features, we introduce MixSignGraph, which represents sign sequences as a group of mixed graphs and designs the following three graph modules for feature extraction, \ie Local Sign Graph (LSG) module, Temporal Sign Graph (TSG) module and Hierarchical Sign Graph (HSG) module. Specifically, the LSG module learns the correlation of intra-frame cross-region features within one frame, \ie focusing on spatial features. The TSG module tracks the interaction of inter-frame cross-region features among adjacent frames, \ie focusing on temporal features. The HSG module aggregates the same-region features from different-granularity feature maps of a frame, \ie focusing on hierarchical features. In addition, to further improve the performance of sign language tasks without gloss annotations, we propose a simple yet counter-intuitive Text-driven CTC Pre-training (TCP) method, which generates pseudo gloss labels from text labels for model pre-training. Extensive experiments conducted on current five public sign language datasets demonstrate the superior performance of the proposed model. Notably, our model surpasses the SOTA models on multiple sign language tasks across several datasets, without relying on any additional cues.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。