研究手势语言翻译中姿态预处理对模型性能的影响。
Exploring Pose-based Sign Language Translation: Ablation Studies and Attention Insights
- 用姿态数据预处理提升翻译模型鲁棒性
- 适当归一化与插值可显著提高准确率
- 发现专用标记能增强模型表现，适合多模态研究者
手势语言翻译已从孤立识别发展为复杂的连续无词翻译系统。本文探讨姿态数据预处理技术（归一化、插值、增强）对翻译性能的影响。采用基于Transformer的架构，改造T5编码器-解码器模型以处理姿态表示。在YouTubeASL和How2Sign数据集上进行大量消融实验，分析不同预处理策略的效果。结果表明，合理的归一化、插值与增强技术能显著提升模型鲁棒性与泛化能力。此外，对模型注意力机制的深入分析揭示：添加专用注册标记可改善整体性能。代码及预处理后的YouTubeASL数据已在GitHub开源。
原文摘要 · Abstract (English)
Sign Language Translation (SLT) has evolved significantly, moving from isolated recognition approaches to complex, continuous gloss-free translation systems. This paper explores the impact of pose-based data preprocessing techniques - normalization, interpolation, and augmentation - on SLT performance. We employ a transformer-based architecture, adapting a modified T5 encoder-decoder model to process pose representations. Through extensive ablation studies on YouTubeASL and How2Sign datasets, we analyze how different preprocessing strategies affect translation accuracy. Our results demonstrate that appropriate normalization, interpolation, and augmentation techniques can significantly improve model robustness and generalization abilities. Additionally, we provide a deep analysis of the model's attentions and reveal interesting behavior suggesting that adding a dedicated register token can improve overall model performance. We publish our code on our GitHub repository, including the preprocessed YouTubeASL data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。