提出新自监督框架,提升手语识别的特征区分能力。
SSL-SLR: Self-Supervised Representation Learning for Sign Language Recognition
- 用自由负样本对缓解手语动作相似导致的混淆问题。
- 在多个数据集上比现有方法准确率显著提升。
- 适合缺乏标注数据的手语识别场景使用。
手语识别(SLR)旨在从视频中识别手语动作。由于标注数据稀缺,对比学习等无监督方法成为主流。它们通过拉近同一实例的增强版本(正样本对),推远不同实例(负样本对)来学习有意义表征。但在手语识别中,视频中只有部分区域真正有助于识别;同时,不同手语间存在共享动作,使负样本高度相似,导致难以区分。这使得现有对比方法学到的特征缺乏判别力,下游任务表现差。为此,本文提出一种自监督学习框架,包含两项关键改进:(i) 新的自由负样本对自监督方法;(ii) 新的数据增强技术。该框架在线性评估、半监督学习及跨语言迁移任务中均显著优于多种对比与自监督方法。
原文摘要 · Abstract (English)
Sign language recognition (SLR) is a machine learning task aiming to identify signs in videos. Due to the scarcity of annotated data, unsupervised methods like contrastive learning have become promising in this field. They learn meaningful representations by pulling positive pairs (two augmented versions of the same instance) closer and pushing negative pairs (different from the positive pairs) apart. In SLR, in a sign video, only certain parts provide information that is truly useful for its recognition. Applying contrastive methods to SLR raises two issues: (i) contrastive learning methods treat all parts of a video in the same way, without taking into account the relevance of certain parts over others; (ii) shared movements between different signs make negative pairs highly similar, complicating sign discrimination. These issues lead to learning non-discriminative features for sign recognition and poor results in downstream tasks. In response, this paper proposes a self-supervised learning framework designed to learn meaningful representations for SLR. This framework consists of two key components designed to work together: (i) a new self-supervised approach with free-negative pairs; (ii) a new data augmentation technique. This approach shows a considerable gain in accuracy compared to several contrastive and self-supervised methods, across linear evaluation, semi-supervised learning, and transferability between sign languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。