arXiv:2606.11925cs.CVcs.LG2026-06

用大模型生成手语视频-文本对,无需额外标注即可提升翻译效果。

Corpus Augmentation for Sign Language Translation via LLM-Guided Video Stitching

论文配图:Corpus Augmentation for Sign Language Translation via LLM-Guided Video Stitching
图 1 · 摘自论文原文
  • 基于CTC对齐提取手势片段,用大模型生成新词句组合并拼接成视频。
  • 在相同框架下提升2.92 BLEU-4,显著优于现有方法。
  • 适合需要数据增强的SLT研究者,尤其关注无监督训练场景。

手语翻译(SLT)将手语视频转换为口语文本,对提升无障碍沟通具有重要意义。尽管大规模弱对齐数据集支持了预训练,且无词典方法减少了人工标注依赖,但高质量的平行手语视频-文本对仍稀缺,制约了长尾词汇和新表达形式的泛化能力。本文提出一种无需额外人工标注、外部手语视频库或生成式视频模型的语料扩充方法:利用现有带词典标注的训练数据,通过CTC强制对齐提取单个词的视频片段,借助基于语料锚定的大语言模型生成新的词-句配对,并通过随机采样句子与片段分配合成新序列。生成的合成RGB视频-文本对在下游训练中具备架构无关性,可直接用于基于RGB的SLT模型,也可经管道转换为姿态或特征表示。在相同条件下重评五种近期无词典方法,最大增益仅为0.98 BLEU-4;而本方法在相同框架下实现+2.92 BLEU-4提升,无需修改模型结构或训练流程。此外,我们发现合成数据虽改善目标指标,却损害视觉-语言预训练性能,且基于L2准则优化片段过渡平滑性反而有害,建议突变边界可能起到隐式正则化作用。代码已开源:https://github.com/robizso/slt-datagen。

原文摘要 · Abstract (English)

Sign language translation (SLT) converts sign language video into spoken language text and holds significant promise for improving accessibility and enabling communication between signing and non-signing communities. While large weakly-aligned datasets have enabled pre-training at scale and gloss-free methods have reduced reliance on expert annotation, high-quality parallel sign video-text pairs for fine-tuning remain scarce, limiting generalisation on long-tail vocabulary and unseen constructions. We propose a corpus augmentation approach that requires no additional human annotation, external sign-language video corpora, or generative video models, relying only on the existing gloss-annotated training corpus and an LLM for sentence generation: per-gloss clips are extracted from training videos via CTC forced-alignment, novel gloss-sentence pairs are generated by a corpus-anchored LLM, and synthetic sequences are assembled through random sentence sampling and clip assignment. The resulting synthetic RGB video-text pairs are architecture-agnostic at the downstream training stage and can be consumed directly by RGB-based SLT models, or converted into pose or feature representations by pipelines that derive such inputs from video. Sincan et al. re-evaluated five recent gloss-free methods under strictly identical conditions; the largest verified gain over the GFSLT-VLP baseline was only 0.98 BLEU-4. Our augmentation, applied within the same framework, achieves +2.92 BLEU-4 without any change to architecture or training protocol. We further identify that synthetic data harms vision-language pretraining despite improving its objectives, and that optimising clip transitions for visual smoothness is counter-productive under L2-based criteria; we propose that abrupt boundaries may act as a form of implicit regularisation. Code is available at https://github.com/robizso/slt-datagen.

手语翻译数据增强大模型视频生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。