用大模型自动评估无性别翻译，更准更省事。
An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation
- 让大模型先分析短语再判断整句，提升评估精度。
- 在五种语言上测试，所有模型表现均优于传统方法。
- 无需额外数据，可轻松扩展到新语言，适合研究者使用。
无性别翻译（GNT）旨在源文本未明确提及人物性别时，避免表达其性别。当前的自动化评估方法受限于单语分类器，无法考虑源句信息，且需针对每种语言单独训练数据和微调，难以扩展。本文探索使用大语言模型（LLMs）作为GNT评估工具，提出两种提示策略：一种仅生成句子级判断；另一种采用类思维链方式，先进行短语级标注，再做出句子级判断。在多种语言、五种模型（含开源与专有模型）上的实验证明，LLMs可有效评估GNT。尤其发现，先进行短语级标注再判断的策略显著提升各模型准确率，提供了一种更优、更易扩展的解决方案。
原文摘要 · Abstract (English)
Gender-neutral translation (GNT) aims to avoid expressing the gender of human referents when the source text lacks explicit cues about the gender of those referents. Evaluating GNT automatically is particularly challenging, with current solutions being limited to monolingual classifiers. Such solutions are not ideal because they do not factor in the source sentence and require dedicated data and fine-tuning to scale to new languages. In this work, we address such limitations by investigating the use of large language models (LLMs) as evaluators of GNT. Specifically, we explore two prompting approaches: one in which LLMs generate sentence-level assessments only, and another, akin to a chain-of-thought approach, where they first produce detailed phrase-level annotations before a sentence-level judgment. Through extensive experiments on multiple languages with five models, both open and proprietary, we show that LLMs can serve as evaluators of GNT. Moreover, we find that prompting for phrase-level annotations before sentence-level assessments consistently improves the accuracy of all models, providing a better and more scalable alternative to current solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。