跨语言通用对齐手语视频与字幕,高效准确。
Segment, Embed, and Align: A Universal Recipe for Aligning Subtitles to Signing
- 分段-嵌入-对齐三步法,用预训练模型提取手语特征。
- 四数据集上达到顶尖性能,支持小时级视频分钟内完成对齐。
- 无需重新训练,适配小词典到大规模语料,开源可用。
本研究旨在开发一种通用方法,将字幕(即带有时间戳的口语文本)与连续手语视频对齐。以往方法通常依赖特定语言或数据集的端到端训练,泛化能力有限。本文提出分段-嵌入-对齐(SEA)框架,仅需两个预训练模型:首个模型将视频帧序列分割为单个手势,第二个模型将每个手势视频片段嵌入与文本共享的潜在空间。随后通过轻量级动态规划算法完成对齐,可在普通CPU上于一分钟内处理长达一小时的视频。SEA灵活可扩展,适用于从小型词典到大型连续语料的各种场景。在四个手语数据集上的实验表明,其对齐性能达到当前最优水平,展现出生成高质量双语数据以推动手语处理研究的巨大潜力。SEA代码与模型已公开。
原文摘要 · Abstract (English)
The goal of this work is to develop a universal approach for aligning subtitles (i.e., spoken language text with corresponding timestamps) to continuous sign language videos. Prior approaches typically rely on end-to-end training tied to a specific language or dataset, which limits their generality. In contrast, our method Segment, Embed, and Align (SEA) provides a single framework that works across multiple languages and domains. SEA leverages two pretrained models: the first to segment a video frame sequence into individual signs and the second to embed the video clip of each sign into a shared latent space with text. Alignment is subsequently performed with a lightweight dynamic programming procedure that runs efficiently on CPUs within a minute, even for hour-long episodes. SEA is flexible and can adapt to a wide range of scenarios, utilizing resources from small lexicons to large continuous corpora. Experiments on four sign language datasets demonstrate state-of-the-art alignment performance, highlighting the potential of SEA to generate high-quality parallel data for advancing sign language processing. SEA's code and models are openly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。