用大模型自动生成手语中间表示,无需人工标注即可实现高效手语翻译。
Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation
- 用大模型根据文本生成初始手语词(伪词),再通过弱监督调整顺序以对齐视频
- 在两个基准上超越现有无词框架,接近有词标注方法的性能
- 适合缺乏手语标注数据的研究者,推动手语翻译可扩展性发展
手语翻译旨在将手语视频映射为口语文本。常用方法依赖专家标注的词(gloss)作为中间表示,将任务分解为视频到词识别与词到文本翻译。但该范式严重依赖人工标注词,成本高且现有数据集极少提供,限制了可扩展性。为此,我们提出一种无需人工词标注的伪词生成框架,保留结构化中间表示。具体地,利用少量文本-词示例,通过上下文学习让大语言模型(LLM)生成初步手语词;为进一步提升伪词与视频手势序列的对应关系,采用弱监督学习过程修正伪词顺序,增强对齐。此重排支持引入辅助对齐目标,并可使用高效的连接时序分类(CTC)损失进行监督。我们通过三阶段训练流程,逐步缩小手语与口语之间的模态差距,训练包含视觉编码器和翻译器的SLT模型。尽管方法简单,但在两个手语翻译基准上优于先前无词框架,性能接近基于词的方法。
原文摘要 · Abstract (English)
Sign Language Translation (SLT) aims to map sign language videos to spoken language text. A common approach relies on gloss annotations as an intermediate representation, decomposing SLT into two sub-tasks: video-to-gloss recognition and gloss-to-text translation. While effective, this paradigm depends on expert-annotated gloss labels, which are costly and rarely available in existing datasets, limiting its scalability. To address this challenge, we propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses while preserving the structured intermediate representation. Specifically, we prompt a Large Language Model (LLM) with a few example text-gloss pairs using in-context learning to produce draft sign glosses from spoken language text. To enhance the correspondence between LLM-generated pseudo glosses and the sign sequences in video, we correct the ordering in the pseudo glosses for better alignment via a weakly supervised learning process. This reordering facilitates the incorporation of auxiliary alignment objectives, and allows for the use of efficient supervision via a Connectionist Temporal Classification (CTC) loss. We train our SLT mode, which consists of a vision encoder and a translator, through a three-stage pipeline, which progressively narrows the modality gap between sign language and spoken language. Despite its simplicity, our approach outperforms previous state-of-the-art gloss-free frameworks on two SLT benchmarks and achieves competitive results compared to gloss-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。