用符号参数提示提升低资源手语教学生成效果
Prompting with Sign Parameters for Low-resource Sign Language Instruction Generation
- 将手形、动作、方向等手语参数融入提示词，增强指令结构化
- 构建首个孟加拉手语教学数据集BdSLIG，解决低资源难题
- 适合关注残障包容性AI与小语种手语技术的研究者
手语为聋哑人群体提供双向沟通渠道，但多数手语在人工智能领域仍属低资源状态。手语教学生成（SLIG）旨在生成步骤式文本指令，帮助非手语使用者模仿学习手语动作，促进双向互动。本文提出首个孟加拉手语教学数据集BdSLIG，用于评估视觉语言模型（VLMs）在低资源手语生成任务及长尾视觉概念上的表现，因孟加拉手语极可能未出现在VLM预训练数据中。为提升零样本性能，我们引入符号参数注入（SPI）提示方法，将手形、运动、方向等标准手语参数直接嵌入文本提示。通过在提示中显式包含手语参数，生成的指令更具结构化和可复现性，优于传统自由文本提示。本工作有望推动低资源社区手语学习系统的包容性发展。
原文摘要 · Abstract (English)
Sign Language (SL) enables two-way communication for the deaf and hard-of-hearing community, yet many sign languages remain under-resourced in the AI space. Sign Language Instruction Generation (SLIG) produces step-by-step textual instructions that enable non-SL users to imitate and learn SL gestures, promoting two-way interaction. We introduce BdSLIG, the first Bengali SLIG dataset, used to evaluate Vision Language Models (VLMs) (i) on under-resourced SLIG tasks, and (ii) on long-tail visual concepts, as Bengali SL is unlikely to appear in the VLM pre-training data. To enhance zero-shot performance, we introduce Sign Parameter-Infused (SPI) prompting, which integrates standard SL parameters, like hand shape, motion, and orientation, directly into the textual prompts. Subsuming standard sign parameters into the prompt makes the instructions more structured and reproducible than free-form natural text from vanilla prompting. We envision that our work would promote inclusivity and advancement in SL learning systems for the under-resourced communities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。