提出自适应Transformer架构,提升手语翻译的精度与效率。
ADAT: Time-Series-Aware Adaptive Transformer Architecture for Sign Language Translation
- 引入门控机制动态加权特征,增强细粒度时序建模
- 在两个数据集上实现最高8.7%的翻译准确率提升
- 适合需要高精度手语识别的应用场景
当前手语机器翻译系统依赖手势、面部表情和身体姿态识别,结合自然语言处理将手语转换为文本。近期方法采用Transformer建模长程依赖,但难以捕捉高帧率下手势间的精细短程时序关系,且计算复杂度高,训练效率低。为此,我们提出自适应Transformer(ADAT),通过门控机制实现特征提取增强与自适应加权,突出上下文相关特征,降低训练开销并保持翻译精度。为评估性能,我们构建了首个公开的医疗领域美国手语数据集MedASL。在手语→词元→文本实验中,ADAT相比编码器-解码器Transformer,在PHOENIX14T上提升BLEU-4 0.1%,训练时间减少14.33%;在MedASL上减少3.24%。在手语→文本实验中,于PHOENIX14T上准确率提升8.7%,训练时间减少2.8%;在MedASL上准确率提升4.7%,训练速度加快7.17%。相较于仅编码器或仅解码器基线,尽管因双流结构最慢达12.1%,其准确率仍至少高出6.8%。
原文摘要 · Abstract (English)
Current sign language machine translation systems rely on recognizing hand movements, facial expressions and body postures, and natural language processing, to convert signs into text. Recent approaches use Transformer architectures to model long-range dependencies via positional encoding. However, they lack accuracy in recognizing fine-grained, short-range temporal dependencies between gestures captured at high frame rates. Moreover, their high computational complexity leads to inefficient training. To mitigate these issues, we propose an Adaptive Transformer (ADAT), which incorporates components for enhanced feature extraction and adaptive feature weighting through a gating mechanism to emphasize contextually relevant features while reducing training overhead and maintaining translation accuracy. To evaluate ADAT, we introduce MedASL, the first public medical American Sign Language dataset. In sign-to-gloss-to-text experiments, ADAT outperforms the encoder-decoder transformer, improving BLEU-4 accuracy by 0.1% while reducing training time by 14.33% on PHOENIX14T and 3.24% on MedASL. In sign-to-text experiments, it improves accuracy by 8.7% and reduces training time by 2.8% on PHOENIX14T and achieves 4.7% higher accuracy and 7.17% faster training on MedASL. Compared to encoder-only and decoder-only baselines in sign-to-text, ADAT is at least 6.8% more accurate despite being up to 12.1% slower due to its dual-stream structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。