arXiv:2508.10687cs.CLcs.AI2025-08

用图神经网络降低手语标注成本,实现更精准的孟加拉手语连续翻译。

Continuous Bangla Sign Language Translation: Mitigating the Expense of Gloss Annotation with the Assistance of Graph

  • 融合Transformer与图结构模型,无需词元标注即可翻译。
  • 在4个数据集上刷新最佳性能,最高提升BLEU-4达4.01点。
  • 首次为BornilDB v1.0建立基准,推动无障碍交流研究。

全球数百万聋哑人士受听力障碍影响,手语是其重要沟通方式。但在以口语为主的社会中,手语常被低估,导致沟通障碍与社会排斥。本研究提出连续孟加拉手语翻译方法,通过融合Transformer与图卷积网络(STGCN-LSTM),实现无词元标注的高效翻译。该方法在RWTH-PHOENIX-2014T、CSL-Daily、How2Sign和BornilDB v1.0四个数据集上均取得新最优结果,相比GASLT和slt_how2sign等基线模型,分别提升BLEU-4 4.01、2.07和0.5分。首次在BornilDB v1.0上建立基准,强调无词元翻译对提升聋哑人群沟通可及性的关键作用。

原文摘要 · Abstract (English)

Millions of individuals worldwide are affected by deafness and hearing impairment. Sign language serves as a sophisticated means of communication for the deaf and hard of hearing. However, in societies that prioritize spoken languages, sign language often faces underestimation, leading to communication barriers and social exclusion. The Continuous Bangla Sign Language Translation project aims to address this gap by enhancing translation methods. While recent approaches leverage transformer architecture for state-of-the-art results, our method integrates graph-based methods with the transformer architecture. This fusion, combining transformer and STGCN-LSTM architectures, proves more effective in gloss-free translation. Our contributions include architectural fusion, exploring various fusion strategies, and achieving a new state-of-the-art performance on diverse sign language datasets, namely RWTH-PHOENIX-2014T, CSL-Daily, How2Sign, and BornilDB v1.0. Our approach demonstrates superior performance compared to current translation outcomes across all datasets, showcasing notable improvements of BLEU-4 scores of 4.01, 2.07, and 0.5, surpassing those of GASLT, GASLT and slt_how2sign in RWTH-PHOENIX-2014T, CSL-Daily, and How2Sign, respectively. Also, we introduce benchmarking on the BornilDB v1.0 dataset for the first time. Our method sets a benchmark for future research, emphasizing the importance of gloss-free translation to improve communication accessibility for the deaf and hard of hearing.

手语翻译图神经网络无标注

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。