arXiv:2411.05172cs.CL2024-11ICLR被引 3

提出可学习的隐含程度度量方法,无需参考文本即可量化语句隐含性。

ImpScore: A Learnable Metric For Quantifying The Implicitness Level of Sentence

  • 基于语义与语用差异定义隐含性,构建无参考的可解释回归模型。
  • 在对比学习数据集上训练,与人类判断高度相关(皮尔逊相关系数达0.83)。
  • 适用于检测模型对隐含话语的理解缺陷,尤其适合评估仇恨言论识别系统。

处理隐含语言对于实现自然语言处理系统的精准理解及与用户自然交互至关重要。然而,缺乏准确衡量语言隐含程度的度量标准,严重制约了对模型理解能力的深度评估。本文提出一种无需外部参考的标量度量方法ImpScore,基于传统语言学原理,将‘隐含性’定义为语义意义与语用解释之间的偏离。通过在精心构建的(隐含句,显式句)成对数据集上采用成对对比学习训练一个可解释的回归模型,实现该度量的可操作化。通过用户研究在分布外数据上对比其评估结果与人工判断,验证了其准确性及与人类评价的强相关性(皮尔逊相关系数0.83)。此外,将ImpScore应用于仇恨言论检测数据集,揭示当前大模型在理解高度隐含内容方面的显著不足。代码已开源:https://github.com/audreycs/ImpScore。

原文摘要 · Abstract (English)

Handling implicit language is essential for natural language processing systems to achieve precise text understanding and facilitate natural interactions with users. Despite its importance, the absence of a metric for accurately measuring the implicitness of language significantly constrains the depth of analysis possible in evaluating models' comprehension capabilities. This paper addresses this gap by developing a scalar metric that quantifies the implicitness level of language without relying on external references. Drawing on principles from traditional linguistics, we define "implicitness" as the divergence between semantic meaning and pragmatic interpretation. To operationalize this definition, we introduce ImpScore, a reference-free metric formulated through an interpretable regression model. This model is trained using pairwise contrastive learning on a specially curated dataset consisting of (implicit sentence, explicit sentence) pairs. We validate ImpScore through a user study that compares its assessments with human evaluations on out-of-distribution data, demonstrating its accuracy and strong correlation with human judgments. Additionally, we apply ImpScore to hate speech detection datasets, illustrating its utility and highlighting significant limitations in current large language models' ability to understand highly implicit content. Our metric is publicly available at https://github.com/audreycs/ImpScore.

隐含性度量自然语言理解可解释模型评估指标

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。