arXiv:2512.08094cs.CL2025-12ACL被引 2

跨语言通用对齐手语视频与字幕,高效准确。

Segment, Embed, and Align: A Universal Recipe for Aligning Subtitles to Signing

  • 分段-嵌入-对齐三步法,用预训练模型提取手语特征。
  • 四数据集上达到顶尖性能,支持小时级视频分钟内完成对齐。
  • 无需重新训练,适配小词典到大规模语料,开源可用。

本研究旨在开发一种通用方法,将字幕(即带有时间戳的口语文本)与连续手语视频对齐。以往方法通常依赖特定语言或数据集的端到端训练,泛化能力有限。本文提出分段-嵌入-对齐(SEA)框架,仅需两个预训练模型:首个模型将视频帧序列分割为单个手势,第二个模型将每个手势视频片段嵌入与文本共享的潜在空间。随后通过轻量级动态规划算法完成对齐,可在普通CPU上于一分钟内处理长达一小时的视频。SEA灵活可扩展,适用于从小型词典到大型连续语料的各种场景。在四个手语数据集上的实验表明,其对齐性能达到当前最优水平,展现出生成高质量双语数据以推动手语处理研究的巨大潜力。SEA代码与模型已公开。

原文摘要 · Abstract (English)

The goal of this work is to develop a universal approach for aligning subtitles (i.e., spoken language text with corresponding timestamps) to continuous sign language videos. Prior approaches typically rely on end-to-end training tied to a specific language or dataset, which limits their generality. In contrast, our method Segment, Embed, and Align (SEA) provides a single framework that works across multiple languages and domains. SEA leverages two pretrained models: the first to segment a video frame sequence into individual signs and the second to embed the video clip of each sign into a shared latent space with text. Alignment is subsequently performed with a lightweight dynamic programming procedure that runs efficiently on CPUs within a minute, even for hour-long episodes. SEA is flexible and can adapt to a wide range of scenarios, utilizing resources from small lexicons to large continuous corpora. Experiments on four sign language datasets demonstrate state-of-the-art alignment performance, highlighting the potential of SEA to generate high-quality parallel data for advancing sign language processing. SEA's code and models are openly available.

手语对齐多语言视频理解开源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。