用T5模型实现沙特手语翻译，预训练提升三倍效果
Saudi Sign Language Translation Using T5
- 用T5模型结合新数据集做沙特手语翻译
- 在YouTubeASL上预训练使BLEU-4提升约3倍
- 适合研究手语翻译与跨语言迁移的学者
本文探讨使用T5模型进行沙特手语（SSL）翻译，并构建了一个包含三个挑战性测试协议的新数据集，支持在多种场景下的全面评估。该数据集还捕捉了沙特手语的独特特征，如面部遮挡，给手势识别和翻译带来挑战。实验中，我们比较了在YouTubeASL数据集上预训练的T5模型与直接在SSL数据集上训练的模型性能。结果表明，在YouTubeASL上预训练显著提升模型表现（约3倍于BLEU-4），显示出手语模型中跨语言迁移的有效性。研究强调利用大规模美国手语数据提升沙特手语翻译的潜力，并为开发更高效的手语翻译系统提供洞见。代码已公开于GitHub仓库。
原文摘要 · Abstract (English)
This paper explores the application of T5 models for Saudi Sign Language (SSL) translation using a novel dataset. The SSL dataset includes three challenging testing protocols, enabling comprehensive evaluation across different scenarios. Additionally, it captures unique SSL characteristics, such as face coverings, which pose challenges for sign recognition and translation. In our experiments, we investigate the impact of pre-training on American Sign Language (ASL) data by comparing T5 models pre-trained on the YouTubeASL dataset with models trained directly on the SSL dataset. Experimental results demonstrate that pre-training on YouTubeASL significantly improves models' performance (roughly $3\times$ in BLEU-4), indicating cross-linguistic transferability in sign language models. Our findings highlight the benefits of leveraging large-scale ASL data to improve SSL translation and provide insights into the development of more effective sign language translation systems. Our code is publicly available at our GitHub repository.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。