通过微调特定参数提升机器翻译质量，无需预先定义优质数据集。
Optimal Corpus Aware Training for Neural Machine Translation
- 仅微调少量语料相关参数，保持模型主体不变
- 在英中、英德翻译任务上分别提升3.6和1.8点chrF
- 对超参数不敏感，适合实际部署场景
语料感知训练（CAT）通过将语料元信息注入每个训练样本，在文献中被证明有效，通常称为“标记”方法。此类模型能直接从数据中学习语料的质量、领域与细微差异，并可灵活切换推理行为。然而，传统CAT需在训练前预定义高质量数据组，存在误差风险且效率低。本文提出最优语料感知训练（OCAT），在预训练的CAT模型基础上，冻结大部分参数，仅微调少量与语料相关的参数。实验表明，OCAT轻量、抗过拟合，显著提升模型精度。在WMT23英中与英德翻译任务上，分别取得+3.6和+1.8 chrF的提升，优于常规训练。同时，其性能与当前最先进微调方法相当或更优，且对超参数设置更为鲁棒。
原文摘要 · Abstract (English)
Corpus Aware Training (CAT) leverages valuable corpus metadata during training by injecting corpus information into each training example, and has been found effective in the literature, commonly known as the "tagging" approach. Models trained with CAT inherently learn the quality, domain and nuance between corpora directly from data, and can easily switch to different inference behavior. To achieve the best evaluation, CAT models pre-define a group of high quality data before training starts which can be error-prone and inefficient. In this work, we propose Optimal Corpus Aware Training (OCAT), which fine-tunes a CAT pre-trained model by freezing most of the model parameters and only tuning small set of corpus-related parameters. We show that OCAT is lightweight, resilient to overfitting, and effective in boosting model accuracy. We use WMT23 English to Chinese and English to German translation tasks as our test ground and show +3.6 and +1.8 chrF improvement, respectively, over vanilla training. Furthermore, our approach is on-par or slightly better than other state-of-the-art fine-tuning techniques while being less sensitive to hyperparameter settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。