提升手语翻译准确率，通过多粒度对齐机制显式建模视频与文本关联。
VTaMo: Video-Text Alignment Model for Sign Language Translation
- 引入三层次显式对齐：帧级、全局嵌入空间、位置感知对比学习。
- 在四个数据集上达到最优性能，尤其在细粒度帧-词对应上显著提升。
- 适合手语识别与跨模态对齐研究者，代码开源可复现。
手语翻译（SLT）将连续手语视频转换为口语文本。无词汇（gloss-free）方法依赖预训练视觉编码器和语言模型，但仅通过翻译监督实现隐式跨模态对齐。本文提出VTaMo框架，引入三层显式多粒度对齐：(1) 基于可学习空令牌的熵正则化最优传输，实现细粒度帧到词对应；(2) 可学习正交变换结合地球移动距离（EMD），校准嵌入空间几何结构；(3) 位置对齐对比学习，增强词级表示判别性。在Phoenix-2014T、CSL-Daily、How2Sign和OpenASL数据集上的实验表明，该方法持续达到最先进性能，消融实验验证了各组件的互补贡献。代码已公开于https://github.com/junyi2005/vtamo。
原文摘要 · Abstract (English)
Sign language translation (SLT) converts continuous sign videos into spoken language text. Gloss-free approaches leverage pre-trained visual encoders and language models but rely on implicit cross-modal alignment from translation supervision alone. We present VTaMo, a framework that introduces explicit multi-granularity alignment at three levels: (1) local alignment via entropy-regularized optimal transport with a learnable null token for fine-grained frame-to-token correspondences; (2) global alignment via a learnable orthogonal transformation that calibrates embedding space geometry through Earth Mover's Distance; and (3) position-aligned contrastive learning for discriminative token-level representations. Experiments on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL demonstrate consistent state-of-the-art performance, with ablations confirming the complementary contributions of each component. Code is available at https://github.com/junyi2005/vtamo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。