arXiv:2605.31393cs.CLcs.AI2026-05中稿 · CVPR

用大模型生成目标语改写,提升手语翻译泛化能力。

Target-Side Paraphrase Augmentation for Sign Language Translation with Large Language Models

论文配图:Target-Side Paraphrase Augmentation for Sign Language Translation with Large Language Models
图 1 · 摘自论文原文
  • 用大模型对参考句进行语义忠实的改写,扩充训练数据。
  • 在PHOENIX14T上BLEU-4提升至10.33,增强解码器泛化性。
  • 首次将大模型作为评估者,发现语义提升被传统指标低估。

手语翻译受限于成对的视频/文本语料稀缺,以及真实数据中词汇分布的长尾特性。本文研究一种目标侧增强策略:利用大语言模型(如GPT-4o)生成参考口语句的可控改写版本,而手语输入保持不变。具体地,使用GPT-4o生成语义一致的参考句变体,并在两阶段训练流程中,先在增强语料上预训练一个基于姿态的Signformer型Transformer,再在原始参考句上微调。在三个互补的数据集上评估:PHOENIX14T(德语手语,中等词汇多样性)、希腊手语数据集(高度控制、重复性强)、LSA-T(阿根廷手语,自然语料、词汇量大且长尾稀疏)。结果表明,在PHOENIX14T上,该方法使BLEU-4从9.56提升至10.33,证明改写暴露有助于解码器超越记忆性句式。然而,在近饱和的希腊手语基线和极稀疏的LSA-T设置下,单参考句词重叠度已不足以反映真实性能,促使采用语义层面的补充评估。本研究首次探索大模型生成目标侧改写用于手语翻译增强,并首次引入大模型作为评估者(LLM-as-a-Judge)协议,揭示了传统指标低估的语义保真度提升。

原文摘要 · Abstract (English)

Sign language translation (SLT) remains constrained by the limited availability of paired sign-video/text corpora and by the heavy-tailed vocabularies typical of real-world datasets. We study a target-side augmentation strategy in which a large language model (LLM) generates controlled paraphrase variants of the reference spoken-language sentence while the sign input remains unchanged. Concretely, we use GPT-4o to produce semantically faithful variants of the training targets and train a Signformer-style pose-based Transformer under a two-stage schedule: pre-training on the augmented corpus followed by fine-tuning on the original references. We evaluate this strategy on three datasets that span complementary challenges: PHOENIX14T (German Sign Language), a real-world corpus with moderate lexical diversity; the Greek Sign Language Dataset with highly controlled, repetitive recordings; and LSA-T (Argentinian Sign Language), a naturalistic corpus with a large vocabulary and severe long-tail sparsity. This range allows us to characterize precisely when and why target-side augmentation is beneficial. On PHOENIX14T, augmentation improves BLEU-4 from 9.56 to 10.33, demonstrating that paraphrastic exposure helps the decoder generalize beyond memorized reference phrasing. The near-saturated GSL baseline and the extremely sparse LSA-T setting reveal the limits of the approach: in both cases, single-reference lexical overlap metrics are insufficient to capture the full picture, motivating a complementary semantic evaluation. To our knowledge, this is the first study to examine LLM-generated target-side paraphrases as an augmentation mechanism for SLT, and the first to apply an LLM-as-a-Judge evaluation protocol to SLT. This complementary evaluation reveals gains in semantic fidelity that lexical overlap metrics understate.

手语翻译大模型数据增强语义评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。