大模型能零样本识别手语吗？实验发现效果有限但有潜力。
Sign Language Recognition in the Age of LLMs
- 用提示工程测试大模型零样本识别手语，不微调直接推理。
- 开源模型比传统方法差很多，但大厂闭源模型表现显著提升。
- 适合研究多模态对齐与手语识别通用模型的学者参考。
最近的视觉语言模型（VLMs）在多种多模态推理任务中表现出色，这引发了一个问题：这类通用模型能否在无需特定任务训练的情况下解决手语识别等专业视觉识别问题？本文研究了现代VLM在零样本设置下进行孤立手语识别（ISLR）的能力。我们在WLASL300基准上评估了几种开源和专有VLM。实验表明，在仅使用提示的零样本推理下，当前开源VLM仍远落后于传统监督式ISLR分类器。然而后续实验揭示，这些模型已部分捕捉到手势与文本描述之间的视觉-语义对齐关系。更大的专有模型实现了显著更高的准确率，凸显了模型规模与训练数据多样性的关键作用。所有代码均已公开在GitHub上。
原文摘要 · Abstract (English)
Recent Vision Language Models (VLMs) have demonstrated strong performance across a wide range of multimodal reasoning tasks. This raises the question of whether such general-purpose models can also address specialized visual recognition problems such as isolated sign language recognition (ISLR) without task-specific training. In this work, we investigate the capability of modern VLMs to perform ISLR in a zero-shot setting. We evaluate several open-source and proprietary VLMs on the WLASL300 benchmark. Our experiments show that, under prompt-only zero-shot inference, current open-source VLMs remain far behind classic supervised ISLR classifiers by a wide margin. However, follow-up experiments reveal that these models capture partial visual-semantic alignment between signs and text descriptions. Larger proprietary models achieve substantially higher accuracy, highlighting the importance of model scale and training data diversity. All our code is publicly available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。