arXiv:2502.13297cs.CLcs.AI2025-02

标注个体层面语言理解数据时,加入同用户多条文本可降低31.7%的标签错误。

Understanding and Tackling Label Errors in Individual-Level Nature Language Understanding

  • 基于同一用户多条文本标注主观观点,改进传统单文本标注方式。
  • 重新标注后数据标签错误率降至31.7%(立场检测)和23.3%(话题情感)。
  • 大模型在引入个体因素后准确率超87%,适合研究个体视角的语言任务。

自然语言理解(NLU)旨在让机器理解人类语言。部分任务如立场检测与话题情感分析紧密关联个体主观视角,称为个体层面的NLU。以往常被简化为文本级任务,忽略个体因素,导致推理困难且数据标签错误率高。为此,我们提出基于个体因素的新标注规范:整合同一用户的多条文本,综合判断其主观立场后再标注。基于此规范,我们扩展并重标注了立场检测与话题情感分析数据集。发现原始样本错误率高达31.7%和23.3%。进一步使用大语言模型在重标注数据上实验,结果表明,GPT-4o与Llama3-70B在添加个体因素后准确率均超过87%。消融实验证明个体因素有效性。呼吁未来研究在构建此类数据集时纳入个体因素。重标注数据集已公开于https://github.com/24yearsoldstudent/Individual-NLU。

原文摘要 · Abstract (English)

Natural language understanding (NLU) is a task that enables machines to understand human language. Some tasks, such as stance detection and sentiment analysis, are closely related to individual subjective perspectives, thus termed individual-level NLU. Previously, these tasks are often simplified to text-level NLU tasks, ignoring individual factors. This not only makes inference difficult and unexplainable but often results in a large number of label errors when creating datasets. To address the above limitations, we propose a new NLU annotation guideline based on individual-level factors. Specifically, we incorporate other posts by the same individual and then annotate individual subjective perspectives after considering all individual posts. We use this guideline to expand and re-annotate the stance detection and topic-based sentiment analysis datasets. We find that error rates in the samples were as high as 31.7\% and 23.3\%. We further use large language models to conduct experiments on the re-annotation datasets and find that the large language models perform well on both datasets after adding individual factors. Both GPT-4o and Llama3-70B can achieve an accuracy greater than 87\% on the re-annotation datasets. We also verify the effectiveness of individual factors through ablation studies. We call on future researchers to add individual factors when creating such datasets. Our re-annotation dataset can be found at https://github.com/24yearsoldstudent/Individual-NLU

个体层面标签纠错大模型标注规范

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。