测试德语机器翻译对职业性别的偏见，发现多数模型仍存在刻板印象。
Are All Spanish Doctors Male? Evaluating Gender Bias in German Machine Translation
- 基于德语语法性别设计新评测集，平衡性别与职业刻板印象
- 5个主流模型和1个大语言模型测试，多数存在性别偏见
- 大语言模型表现最好，数据与代码已开源
我们提出WinoMTDE，一个用于评估德语机器翻译系统中职业刻板印象和代表性不足的新性别偏见评测集。该方法在arXiv:1906.00591v1基础上扩展至具有语法性别特征的德语。WinoMTDE包含288个德语句子，按性别和刻板印象平衡，并使用德国劳动统计数据进行标注。我们对五个广泛使用的机器翻译系统和一个大语言模型进行了大规模评估。结果表明，大多数模型仍存在持续偏见，而大语言模型表现优于传统系统。数据集和评估代码已公开，地址为https://github.com/michellekappl/mt_gender_german。
原文摘要 · Abstract (English)
We present WinoMTDE, a new gender bias evaluation test set designed to assess occupational stereotyping and underrepresentation in German machine translation (MT) systems. Building on the automatic evaluation method introduced by arXiv:1906.00591v1, we extend the approach to German, a language with grammatical gender. The WinoMTDE dataset comprises 288 German sentences that are balanced in regard to gender, as well as stereotype, which was annotated using German labor statistics. We conduct a large-scale evaluation of five widely used MT systems and a large language model. Our results reveal persistent bias in most models, with the LLM outperforming traditional systems. The dataset and evaluation code are publicly available under https://github.com/michellekappl/mt_gender_german.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。