为手语书写设计专用评估工具，解决手语转写模型评价难题。
signwriting-evaluation: Effective Sign Language Evaluation via SignWriting
- 针对手语书写特点改造通用指标，新增符号距离度量方法。
- 验证不同指标在单个手势与连续手语上的表现差异。
- 适合手语识别与翻译研究者使用，推动手语技术发展。
当前缺乏针对手语书写（SignWriting）的自动化评估指标，严重制约了手语转写与翻译模型的发展。本文提出一套专为SignWriting设计的综合评估体系，包括对标准指标BLEU、chrF的适配，将CLIPScore应用于SignWriting图像，以及一种独特的符号距离度量。针对单个手势与连续手语的不同评价挑战，通过SignBank语料库中的得分分布分析和最近邻搜索进行定性验证。结果揭示各指标的优势与局限，为后续基于SignWriting的研究提供重要参考。代码已开源于https://github.com/sign-language-processing/signwriting-evaluation。
原文摘要 · Abstract (English)
The lack of automatic evaluation metrics tailored for SignWriting presents a significant obstacle in developing effective transcription and translation models for signed languages. This paper introduces a comprehensive suite of evaluation metrics specifically designed for SignWriting, including adaptations of standard metrics such as \texttt{BLEU} and \texttt{chrF}, the application of \texttt{CLIPScore} to SignWriting images, and a novel symbol distance metric unique to our approach. We address the distinct challenges of evaluating single signs versus continuous signing and provide qualitative demonstrations of metric efficacy through score distribution analyses and nearest-neighbor searches within the SignBank corpus. Our findings reveal the strengths and limitations of each metric, offering valuable insights for future advancements using SignWriting. This work contributes essential tools for evaluating SignWriting models, facilitating progress in the field of sign language processing. Our code is available at \url{https://github.com/sign-language-processing/signwriting-evaluation}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。