用英语习语和地名挑战大模型翻译，发现系统表现差强人意。
Killing Two Flies with One Stone: An Attempt to Break LLMs Using English->Icelandic Idioms and Proper Names
- 构建双测试集：习语与专有名词的翻译难题。
- 习语和地名翻译得分偏低，暴露出模型理解缺陷。
- 适合研究低资源语言翻译与文化语义建模的学者参考。
本文介绍了Árni Magnússon研究所团队提交的WMT24测试套件子任务，聚焦英语到冰岛语翻译中的习语和专有名词。直观上且实证表明，习语和专有名词是现代机器翻译系统的重大挑战。我们设计了两个测试集：第一个评估翻译系统对常见英语习语的处理能力，并检验其能否区分习语与字面用法；第二个测试集包含应译为冰岛语外称（并正确变格）的地名，以及男女形式相同但含义不同的冰岛姓名对，错误翻译会同时影响语义与可读性。报告得分普遍较低，尤其在习语和地名方面，显示出显著的改进空间。
原文摘要 · Abstract (English)
This paper presents the submission of the Árni Magnússon Institute's team to the WMT24 test suite subtask, focusing on idiomatic expressions and proper names for the English->Icelandic translation direction. Intuitively and empirically, idioms and proper names are known to be a significant challenge for modern translation models. We create two different test suites. The first evaluates the competency of MT systems in translating common English idiomatic expressions, as well as testing whether systems can distinguish between those expressions and the same phrases when used in a literal context. The second test suite consists of place names that should be translated into their Icelandic exonyms (and correctly inflected) and pairs of Icelandic names that share a surface form between the male and female variants, so that incorrect translations impact meaning as well as readability. The scores reported are relatively low, especially for idiomatic expressions and place names, and indicate considerable room for improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。