arXiv:2503.08042cs.CL2025-03ACL被引 4

用合成数据评估词汇语义变化检测方法,填补历史基准空白。

LSC-Eval: A General Framework to Evaluate Methods for Assessing Dimensions of Lexical Semantic Change Using LLM-Generated Synthetic Data

  • 基于上下文学习生成模拟语义变化的合成数据
  • 验证了特定方法对情感、强度等维度变化的检测能力
  • 适合社会科学研究者用于方法对比与验证

词汇语义变化(LSC)揭示文化与社会动态。然而,由于缺乏历史基准数据集,现有测量方法的有效性尚不明确。为此,我们提出 LSC-Eval,一个三阶段通用评估框架:(1) 利用上下文学习与词典构建可扩展的合成数据生成方法,模拟理论驱动的语义变化;(2) 用这些数据评估计算方法对合成变化的敏感性;(3) 评估其在特定维度与领域中检测变化的适用性。我们以心理学为例,在情感(Sentiment)、强度(Intensity)、广度(Breadth)(SIB)维度上生成变化实例,应用 SIBling 框架进行建模,并测试多种方法的检测能力。结果表明合成基准有效,定制化方法能准确识别 SIB 维度变化,而当前先进模型在情感维度检测上仍存挑战。LSC-Eval 为维度与领域特异性评估 LSC 方法提供了有力工具,尤其适用于社会科学领域。

原文摘要 · Abstract (English)

Lexical Semantic Change (LSC) provides insight into cultural and social dynamics. Yet, the validity of methods for measuring different kinds of LSC remains unestablished due to the absence of historical benchmark datasets. To address this gap, we propose LSC-Eval, a novel three-stage general-purpose evaluation framework to: (1) develop a scalable methodology for generating synthetic datasets that simulate theory-driven LSC using In-Context Learning and a lexical database; (2) use these datasets to evaluate the sensitivity of computational methods to synthetic change; and (3) assess their suitability for detecting change in specific dimensions and domains. We apply LSC-Eval to simulate changes along the Sentiment, Intensity, and Breadth (SIB) dimensions, as defined in the SIBling framework, using examples from psychology. We then evaluate the ability of selected methods to detect these controlled interventions. Our findings validate the use of synthetic benchmarks, demonstrate that tailored methods effectively detect changes along SIB dimensions, and reveal that a state-of-the-art LSC model faces challenges in detecting affective dimensions of LSC. LSC-Eval offers a valuable tool for dimension- and domain-specific benchmarking of LSC methods, with particular relevance to the social sciences.

语义变化合成数据自然语言社会计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。