用大模型生成解释，能高效逼近人类标签分布。
A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI
- 让大模型为少数人工标签生成解释，替代人工标注。
- 生成解释在自然语言推理任务中可达到与真人相当的标签分布估计效果。
- 方法适用于无解释数据集和分布外测试场景，适合大规模标注需求者。
人类标注中的分歧普遍存在，可通过人类判断分布（HJD）捕捉。近期研究发现，解释信息有助于理解人类标签差异，且大语言模型（LLM）仅需少量人工提供的标签-解释对即可近似 HJD。然而，为每个标签收集解释仍耗时。本文探讨是否可用 LLM 替代人工生成解释以近似 HJD。具体地，我们使用 LLM 作为标注者，为若干给定的人工标签生成模型解释，并测试不同获取与组合方式以逼近人类判断分布。进一步对比了由人与模型生成的解释，并评估自动与人工解释选择策略。实验表明，在自然语言推理（NLI）任务中，当提供人工标签时，大模型生成的解释能获得与人类相当的 HJD 估计效果。更重要的是，该结果可推广至无解释数据集及具有挑战性的分布外测试集。
原文摘要 · Abstract (English)
Disagreement in human labeling is ubiquitous, and can be captured in human judgment distributions (HJDs). Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) and large language models (LLMs) can approximate HJD from a few human-provided label-explanation pairs. However, collecting explanations for every label is still time-consuming. This paper examines whether LLMs can be used to replace humans in generating explanations for approximating HJD. Specifically, we use LLMs as annotators to generate model explanations for a few given human labels. We test ways to obtain and combine these label-explanations with the goal to approximate human judgment distributions. We further compare the resulting human with model-generated explanations, and test automatic and human explanation selection. Our experiments show that LLM explanations are promising for NLI: to estimate HJDs, generated explanations yield comparable results to human's when provided with human labels. Importantly, our results generalize from datasets with human explanations to i) datasets where they are not available and ii) challenging out-of-distribution test sets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。