动态解冻预训练层,提升手语翻译的适应性与准确率。
HATL: Hierarchical Adaptive-Transfer Learning Framework for Sign Language Machine Translation
- 按训练表现逐步动态解冻预训练层,避免过拟合。
- 在三个数据集上,最高提升37.6% BLEU-4得分。
- 适合需要跨语言、跨签名者泛化的手语翻译场景。
手语机器翻译(SLMT)旨在弥合聋人与听人之间的沟通鸿沟。然而,其发展受限于数据稀缺、签名人多样性不足以及手语动作模式与预训练表示之间的巨大领域差异。现有SLMT中的迁移学习方法多为静态策略,常导致过拟合。为此,本文提出分层自适应迁移学习(HATL)框架,根据训练性能动态逐层解冻预训练权重。HATL结合动态解冻、层间学习率衰减与稳定性机制,在保留通用特征的同时有效适应手语特性。我们在Sign2Text和Sign2Gloss2Text任务上评估该方法,采用ST-GCN++作为特征提取器,Transformer与自适应Transformer(ADAT)作为翻译模型。为验证多语言泛化能力,测试覆盖三组数据集:RWTH-PHOENIXWeather-2014(PHOENIX14T)、Isharah与MedASL。实验表明,HATL在各类任务与模型中均优于传统迁移学习,其中ADAT在PHOENIX14T和Isharah上实现15.0%的BLEU-4提升,在MedASL上达37.6%提升。
原文摘要 · Abstract (English)
Sign Language Machine Translation (SLMT) aims to bridge communication between Deaf and hearing individuals. However, its progress is constrained by scarce datasets, limited signer diversity, and large domain gaps between sign motion patterns and pretrained representations. Existing transfer learning approaches in SLMT are static and often lead to overfitting. These challenges call for the development of an adaptive framework that preserves pretrained structure while remaining robust across linguistic and signing variations. To fill this void, we propose a Hierarchical Adaptive Transfer Learning (HATL) framework, where pretrained layers are progressively and dynamically unfrozen based on training performance behavior. HATL combines dynamic unfreezing, layer-wise learning rate decay, and stability mechanisms to preserve generic representations while adapting to sign characteristics. We evaluate HATL on Sign2Text and Sign2Gloss2Text translation tasks using a pretrained ST-GCN++ backbone for feature extraction and the Transformer and an adaptive transformer (ADAT)for translation. To ensure robust multilingual generalization, we evaluate the proposed approach across three datasets: RWTH-PHOENIXWeather-2014 (PHOENIX14T), Isharah, and MedASL. Experimental results show that HATL consistently outperforms traditional transfer learning approaches across tasks and models, with ADAT achieving BLEU-4 improvements of 15.0% on PHOENIX14T and Isharah and 37.6% on MedASL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。