arXiv:2511.08949cs.CL2025-11ACL被引 1

用大模型生成并验证错误解释,提升自然语言推理数据集质量

EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI

  • 用大模型自动生成解释,并验证其合理性以识别标注错误
  • 大模型发现的错误去除后,模型微调效果优于人工发现的错误
  • 在存在标签多样性时,能高效降低人工标注成本

高质量数据集对训练和评估可靠的NLP模型至关重要。在自然语言推理(NLI)任务中,当同一实例存在多个有效标签时,会引发人类标注差异(HLV),使得难以区分标注错误与合理变异。先前的VARIERR框架要求多轮人工标注:第一轮由标注员提供决策解释,第二轮通过有效性判断标记错误。但该方法成本高,可能限制合理标签或解释的覆盖范围。本文提出EVADE框架,利用大语言模型(LLM)生成并验证解释以检测错误。我们对人类与LLM检测的错误进行了全面分析,涵盖分布比较、验证重叠度及对模型微调的影响。实验表明,使用LLM验证可使生成的解释分布更接近人类标注;且从训练数据中移除LLM检测到的错误,比移除人工检测的错误能带来更好的微调性能。这表明,该方法具备规模化错误检测潜力,在减少人工投入的同时提升含标签变异的数据集质量。

原文摘要 · Abstract (English)

High-quality datasets are critical for training and evaluating reliable NLP models. In tasks like natural language inference (NLI), human label variation (HLV) arises when multiple labels are valid for the same instance, making it difficult to separate annotation errors from plausible variation. An earlier framework, VARIERR (Weber-Genzel et al., 2024), asks multiple annotators to explain their label decisions in the first round and flags errors through validity judgments in the second round. However, conducting two rounds of manual annotation is costly and may limit the coverage of plausible labels or explanations. Our study proposes a new framework, EVADE, for generating and validating explanations to detect errors using large language models (LLMs). We perform a comprehensive analysis comparing human- and LLM-detected errors for NLI across distribution comparison, validation overlap, and impact on model fine-tuning. Our experiments demonstrate that LLM validation refines generated explanation distributions to more closely align with human annotations, and that removing LLM-detected errors from training data yields improvements in fine-tuning performance than removing errors identified by human annotators. This highlights the potential to scale error detection, reducing human effort while improving dataset quality under label variation.

自然语言推理大模型应用错误检测数据质量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。