用智能选点替代随机抽样，降低文本生成评估成本。
How to Select Datapoints for Efficient Human Evaluation of NLG Models?
- 基于自动指标方差、输出多样性等策略选最具信息量的样本。
- 只需约70%数据即可达到全量评估效果，节省评估成本。
- 提出仅凭源文本预测样本价值的方法，适用于模型未生成时。
人工评估是文本生成模型的黄金标准，但成本高昂。实践中常随机选取测试集子集进行评估，但随机样本可能无法准确反映模型真实表现，导致资源浪费。为此，本文开发并分析了一系列选择器，旨在选出最具信息量的样本以兼顾评估成本。实验表明，基于自动化指标方差、模型输出多样性或项目反应理论的选择器优于随机选择。我们进一步提出一种在模型输出尚未生成时的蒸馏方法，引入基于源文本的估计器，仅凭源文本预测样本对人工评估的价值。在机器翻译和摘要两个常见任务中验证了所提方法的有效性，结果表明仅需约70%的测试数据即可获得与全量评估相当的结果。
原文摘要 · Abstract (English)
Human evaluation is the gold standard for evaluating text generation models. However, it is expensive. In order to fit budgetary constraints, a random subset of the test data is often chosen in practice for human evaluation. However, randomly selected data may not accurately represent test performance, making this approach economically inefficient for model comparison. Thus, in this work, we develop and analyze a suite of selectors to get the most informative datapoints for human evaluation, taking the evaluation costs into account. We show that selectors based on variance in automated metric scores, diversity in model outputs, or Item Response Theory outperform random selection. We further develop an approach to distill these selectors to the scenario where the model outputs are not yet available. In particular, we introduce source-based estimators, which predict item usefulness for human evaluation just based on the source texts. We demonstrate the efficacy of our selectors in two common NLG tasks, machine translation and summarization, and show that only $\sim$70\% of the test data is needed to produce the same evaluation result as the entire data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。