arXiv:2410.09807cs.CLcs.AI2024-10NAACL被引 3

为情感分析评估引入多答案选项,更公平地衡量模型表现。

Single Ground Truth Is Not Enough: Adding Flexibility to Aspect-Based Sentiment Analysis Evaluation

  • 构建自动化流程,为每个评价项添加多个语义有效的替代词。
  • 多答案测试集使人工一致性提升10%(Kendall's Tau)。
  • 适合评估大模型在细粒度情感分析中的真实能力。

基于方面的情感分析(ABSA)旨在从文本中提取情感及其对应方面与观点词。由于标注的主观性,提取词的表面形式存在差异,导致评估困难。传统方法通常限定真实答案(GT)为单一词,可能低估语义正确但形式不同的预测结果。为此,我们提出一种全自动新流程,通过为方面和观点词添加多个有效替代项,扩展现有评估集。该方法使语言模型评估更公平,显著提升人机一致性(最高达Kendall's Tau提升10%)。实验表明,该扩展数据集能揭示大语言模型在ABSA任务中的真实能力,而这些能力在单答案基准下被掩盖。本工作推动了可扩展、低成本、可复现的弹性评估框架发展,代码与数据集已开源。

原文摘要 · Abstract (English)

Aspect-based sentiment analysis (ABSA) is a challenging task of extracting sentiments along with their corresponding aspects and opinion terms from the text. The inherent subjectivity of span annotation makes variability in the surface forms of extracted terms, complicating the evaluation process. Traditional evaluation methods often constrain ground truths (GT) to a single term, potentially misrepresenting the accuracy of semantically valid predictions that differ in surface form. To address this limitation, we propose a novel and fully automated pipeline that expands existing evaluation sets by adding alternative valid terms for aspect and opinion. Our approach facilitates an equitable assessment of language models by accommodating multiple-answer candidates, resulting in enhanced human agreement compared to single-answer test sets (achieving up to a 10\%p improvement in Kendall's Tau score). Experimental results demonstrate that our expanded evaluation set helps uncover the capabilities of large language models (LLMs) in ABSA tasks, which is concealed by the single-answer GT sets. Consequently, our work contributes to the development of a flexible evaluation framework for ABSA by embracing diverse surface forms to span extraction tasks in a cost-effective and reproducible manner. Our code and dataset is open at https://github.com/dudrrm/zoom-in-n-out-absa.

情感分析评估改进大模型评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。