构建首个基于问答的手语理解评估基准，聚焦语义准确性而非表面匹配。
SLU-2K: A Question-Based Benchmark for Semantic Evaluation of Sign Language Translation
- 提出自动化数据生成流程，构建涵盖7类语义的2350个问答对
- 主流大模型在该任务上表现接近随机，准确率仅56.7%~75.2%
- 适用于评估手语理解系统的真实语义保真度，适合辅助技术研究者
手语翻译（SLT）通常采用BLEU、ROUGE等表层指标评估，仅关注词汇重叠，无法衡量译文是否保留源手语序列的语义。为此，本文转向手语理解（SLU），重点评估系统从输入视频中恢复关键语义的能力，如动作、人物、物体等事实。为此，我们构建了SLU-2K，一个基于PHOENIX-2014T和CSL-Daily数据集的2,350个封闭式问答对数据集。通过自动化数据生成流程，生成覆盖动作、位置、数量、物体、人物、时间、天气共7类问题。我们评估了主流多模态大模型（MLLMs）及两个先进系统（MMSTL、SpaMo），结果显示：MLLMs表现接近随机，而最优系统仍存在显著语义差距，准确率在56.7%至75.2%之间。这表明现有评估协议高估了真实理解能力，未来进展应兼顾流畅性与语义正确性。代码、提示与基准文件已开源。
原文摘要 · Abstract (English)
Sign Language Translation (SLT) is typically evaluated with surface-form metrics such as BLEU and ROUGE, which reward lexical overlap but do not directly measure whether a translation preserves the meaning of the source sign sequence. This is in contrast with the final objective of integrating SLT in assistive technology. In this work, we shift the focus from Sign Language Translation (SLT) to Sign Language Understanding (SLU), with particular emphasis on semantic understanding. Specifically, we evaluate systems based on their ability to correctly recover, from the input video, key semantic aspects of the original sentence, such as actions taking place and facts about people and objects. To enable this evaluation systematically, we propose SLU-2K, a dataset of 2,350 closed-ended video question-answer pairs based on the popular PHOENIX-2014T and CSL-Daily datasets. To obtain SLU-2K, we propose and extensively evaluate an automated data generation pipeline which produces questions across 7 categories, namely actions, locations, numbers, objects, people, time, and weather conditions. We show the potential of SLU-2K by evaluating popular Multimodal Large Language Models (MLLMs) and two representative state-of-the-art systems, MMSTL and SpaMo. Our results show that MLLMs reach near-random performance, highlighting the need for a more systematic integration of SLU in current AI systems. Furthermore, state-of-the-art translation systems carefully fine-tuned on in-domain data still exhibit a substantial semantic gap, with results ranging from 56.7% to 75.2%. These findings suggest that current SLT evaluation protocols overestimate true understanding and that future progress should be measured not only by fluency and n-gram overlap, but also by semantic correctness. Code, prompts, and benchmark files are available at https://github.com/ZenoTsT/SLU-2K
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。