模型训练能自动选出更优的词汇表，提升翻译质量。
Self-Vocabularizing Training for Neural Machine Translation
- 通过自训练迭代生成新词汇表，动态优化翻译用词
- 在相同数据上训练，最高提升1.49 BLEU得分
- 深层模型用词更独特，词汇量减少6-8%
以往词汇学习方法在训练前确定词汇表，依赖统计和熵的假设，忽略了模型训练本身的影响。我们观察到，训练后的翻译模型会倾向于使用与原始BPE词汇表不同的子集，重新用该子集训练可提升性能。本文通过分析自训练过程中词汇和熵的变化揭示这一现象：每轮用模型自身预测生成带标签数据，并据此构建新词汇表。基于此，提出自词汇化训练方法，迭代选择更小、更优的词汇表，在实验中实现最高1.49 BLEU的提升。此外发现，更深的模型架构不仅增加唯一词符使用率，还使词汇量减少6%-8%。
原文摘要 · Abstract (English)
Past vocabulary learning techniques identify relevant vocabulary before training, relying on statistical and entropy-based assumptions that largely neglect the role of model training. Empirically, we observe that trained translation models are induced to use a byte-pair encoding (BPE) vocabulary subset distinct from the original BPE vocabulary, leading to performance improvements when retrained with the induced vocabulary. In this paper, we analyze this discrepancy in neural machine translation by examining vocabulary and entropy shifts during self-training--where each iteration generates a labeled dataset by pairing source sentences with the model's predictions to define a new vocabulary. Building on these insights, we propose self-vocabularizing training, an iterative method that self-selects a smaller, more optimal vocabulary, yielding up to a 1.49 BLEU improvement. Moreover, we find that deeper model architectures lead to both an increase in unique token usage and a 6-8% reduction in vocabulary size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。