用文本对齐的CTC提升手语翻译，解决视频与文字非单调对应问题。
Improvement in Sign Language Translation Using Text CTC Alignment
- 融合CTC与注意力机制，分层编码处理单调与非单调对齐。
- 在RWTH和CSL数据集上超越纯注意力基线，接近顶尖水平。
- 为无词元手语翻译提供新思路，适合关注跨模态对齐的研究者。
当前手语翻译方法多依赖词元监督与连接时序分类（CTC），难以处理手语视频与口语文本间的非单调对齐。本文提出结合联合CTC/注意力机制与迁移学习的新方法。联合机制通过分层编码，在解码阶段融合CTC与注意力，有效应对单调与非单调对齐。迁移学习则缓解视觉与语言模态间的差距。在两大主流基准数据集RWTH-PHOENIX-Weather 2014 T与CSL-Daily上的实验表明，该方法性能接近最先进水平，优于纯注意力基线。此外，本工作为基于文本对齐的无词元手语翻译开辟了新路径。
原文摘要 · Abstract (English)
Current sign language translation (SLT) approaches often rely on gloss-based supervision with Connectionist Temporal Classification (CTC), limiting their ability to handle non-monotonic alignments between sign language video and spoken text. In this work, we propose a novel method combining joint CTC/Attention and transfer learning. The joint CTC/Attention introduces hierarchical encoding and integrates CTC with the attention mechanism during decoding, effectively managing both monotonic and non-monotonic alignments. Meanwhile, transfer learning helps bridge the modality gap between vision and language in SLT. Experimental results on two widely adopted benchmarks, RWTH-PHOENIX-Weather 2014 T and CSL-Daily, show that our method achieves results comparable to state-of-the-art and outperforms the pure-attention baseline. Additionally, this work opens a new door for future research into gloss-free SLT using text-based CTC alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。