研究发现机器翻译性别偏见导致女性用户付出更高修改成本。
What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study
- 通过90人实证实验，测量人工修正性别翻译的耗时与难度
- 女性相关文本需多花47%时间、1.8倍技术努力修正
- 呼吁用真实用户数据评估模型社会影响，而非仅依赖自动指标
机器翻译中的性别偏见被广泛认为可能对个人和社会造成伤害，但现有进展极少涉及终端用户，也缺乏对其实际影响的了解。当前评估多依赖自动方法，难以反映下游真实影响。本文开展大规模以人为中心的研究，探究性别偏差是否带来可量化的实际代价。我们收集了90名参与者在多个数据集、语言和用户类型下的后编辑行为数据，要求其修正机器翻译中的性别错误。结果显示，修正女性相关文本所需的技术投入和时间显著高于男性，平均耗时增加47%，技术努力度提升1.8倍，对应更高的经济成本。而现有偏见度量指标未能捕捉到这些差异。研究呼吁采用以人为中心的方法，以更准确地揭示偏见的社会影响。
原文摘要 · Abstract (English)
Gender bias in machine translation (MT) is recognized as an issue that can harm people and society. And yet, advancements in the field rarely involve people, the final MT users, or inform how they might be impacted by biased technologies. Current evaluations are often restricted to automatic methods, which offer an opaque estimate of what the downstream impact of gender disparities might be. We conduct an extensive human-centered study to examine if and to what extent bias in MT brings harms with tangible costs, such as quality of service gaps across women and men. To this aim, we collect behavioral data from 90 participants, who post-edited MT outputs to ensure correct gender translation. Across multiple datasets, languages, and types of users, our study shows that feminine post-editing demands significantly more technical and temporal effort, also corresponding to higher financial costs. Existing bias measurements, however, fail to reflect the found disparities. Our findings advocate for human-centered approaches that can inform the societal impact of bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。