用极少量参数让CLIP模型高效识别手语，提升准确率。
CLIP-SLA: Parameter-Efficient CLIP Adaptation for Continuous Sign Language Recognition
- 在CLIP视觉编码器中加入轻量适配模块，实现参数高效微调。
- 在4个数据集上超越多个主流模型，参数量减少超90%。
- 适合资源有限但需高精度手语识别的场景，如移动端应用。
连续手语识别（CSLR）旨在解析视频中连续的手语手势序列。本文提出一种基于CLIP模型的参数高效适配框架——CLIP-SLA，通过参数高效微调（PEFT）将预训练的CLIP视觉编码器应用于手语任务。设计了两种变体：SLA-Adapter与SLA-LoRA，分别在CLIP视觉编码器中引入适配模块，仅需极少可训练参数即可完成微调。在Phoenix2014、Phoenix2014-T、CSL-Daily和Isharah-500四个数据集上的实验表明，两种变体均优于多个现有先进模型，且参数量显著降低。大量消融实验验证了方法的有效性与灵活性，适用于不同视觉-语言模型。研究结果展示了大规模预训练模型在可扩展、高效手语识别中的潜力，为未来手语理解技术发展提供新路径。
原文摘要 · Abstract (English)
Continuous sign language recognition (CSLR) focuses on interpreting and transcribing sequences of sign language gestures in videos. In this work, we propose CLIP sign language adaptation (CLIP-SLA), a novel CSLR framework that leverages the powerful pre-trained visual encoder from the CLIP model to sign language tasks through parameter-efficient fine-tuning (PEFT). We introduce two variants, SLA-Adapter and SLA-LoRA, which integrate PEFT modules into the CLIP visual encoder, enabling fine-tuning with minimal trainable parameters. The effectiveness of the proposed frameworks is validated on four datasets: Phoenix2014, Phoenix2014-T, CSL-Daily, and Isharah-500, where both CLIP-SLA variants outperformed several SOTA models with fewer trainable parameters. Extensive ablation studies emphasize the effectiveness and flexibility of the proposed methods with different vision-language models for CSLR. These findings showcase the potential of adapting large-scale pre-trained models for scalable and efficient CSLR, which pave the way for future advancements in sign language understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。