发现翻译评估中存在跨方向记忆污染，模型会无中生有地提升未训练方向的翻译表现。
When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation
- 用FLORES-200诊断多语言模型，发现Bloomz在目标语言侧记忆了参考译文
- 未训练方向的翻译性能因目标侧记忆被虚假提升，出现跨方向污染
- 替换命名实体可稳定降低BLEU值，是检测模型记忆的有效方法
大型语言模型（LLMs）可能在基准测试中产生过拟合，导致分数虚高，掩盖了记忆而非泛化的能力。在多语言场景下，这种记忆甚至会传播至“未污染”语言。本文以FLORES-200翻译基准为诊断工具，研究两个7-8B参数量的指令微调多语言LLM：Bloomz（在FLORES上训练）和作为对照的Llama。结果确认Bloomz存在FLORES污染，并证明机器翻译的污染可呈跨方向性——由于目标侧记忆，未见过的翻译方向性能被人为提升。进一步分析显示，即使对源端进行改写或实体替换等扰动，记忆内容仍能持续被召回。但替换命名实体后，BLEU得分始终下降，表明该方法可有效探测污染模型中的记忆行为。
原文摘要 · Abstract (English)
Large language models (LLMs) can be benchmark-contaminated, resulting in inflated scores that mask memorization as generalization, and in multilingual settings, this memorization can even transfer to "uncontaminated" languages. Using the FLORES-200 translation benchmark as a diagnostic, we study two 7-8B instruction-tuned multilingual LLMs: Bloomz, which was trained on FLORES, and Llama as an uncontaminated control. We confirm Bloomz's FLORES contamination and demonstrate that machine translation contamination can be cross-directional, artificially boosting performance in unseen translation directions due to target-side memorization. Further analysis shows that recall of memorized references often persists despite various source-side perturbation efforts like paraphrasing and named entity replacement. However, replacing named entities leads to a consistent decrease in BLEU, suggesting an effective probing method for memorization in contaminated models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。