测试大模型能否通过上下文理解习语含义，发现其常因依赖表面线索而失败。
Rolling the DICE on Idiomaticity: How LLMs Fail to Grasp Context
- 构建受控对比数据集，检验模型是否真能利用上下文解析习语
- 模型在高概率句子中表现更好，对上下文依赖强的习语识别率下降
- 习语搭配频率影响模型表现，说明其依赖统计规律而非深层理解
人类理解习语依赖于上下文语境、语言内在特征（如出现频率）及说话者熟悉度。尽管大语言模型在习语识别任务中表现良好，这种成功可能源于现有数据集中的推理捷径。为此，我们构建了一个新型受控对比数据集，用于检验模型是否能有效利用上下文区分习语含义。同时研究了搭配频率与句子概率对模型性能的影响。结果表明，当需要关注上下文时，模型往往无法正确解析习语意义；在高概率句子中表现更优，且搭配频率也显著影响识别效果。代码与数据集已公开。
原文摘要 · Abstract (English)
Human processing of idioms relies on understanding the contextual sentences in which idioms occur, as well as language-intrinsic features such as frequency and speaker-intrinsic factors like familiarity. While LLMs have shown high performance on idiomaticity detection tasks, this success may be attributed to reasoning shortcuts in existing datasets. To this end, we construct a novel, controlled contrastive dataset designed to test whether LLMs can effectively use context to disambiguate idiomatic meaning. Additionally, we explore how collocational frequency and sentence probability influence model performance. Our findings reveal that LLMs often fail to resolve idiomaticity when it is required to attend to the surrounding context, and that models perform better on sentences that have higher likelihood. The collocational frequency of expressions also impacts performance. We make our code and dataset publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。