arXiv:2508.13365cs.CL2025-08

用推理提升语言模型识别成语能力,发现大模型表现更好。

Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection

  • 让模型生成推理链来判断表达是否为成语。
  • 70B大模型推理后准确率提升,小模型效果有限。
  • 给小模型提供成语解释可部分弥补能力不足。

近年来,具备推理能力的大型语言模型在需要逻辑步骤的任务中表现优异。本研究探讨推理能力对成语识别任务的影响,并分析模型规模的作用。我们评估了从1.5B到70B参数的DeepSeek-R1系列开源模型在四个成语识别数据集上的表现。结果发现,推理对性能的提升不如预期显著:中小模型(1.5B–7B)通过链式思维(CoT)推理后性能有所提升,但未达基线模型水平;而14B、32B和70B的大模型仅获得小幅改善。深入分析显示,大模型能正确理解并定义成语,而小模型常无法输出真实含义。因此,我们在小模型提示中加入成语定义,实验表明该方法在某些情况下可有效提升性能。

原文摘要 · Abstract (English)

The recent trend towards utilisation of reasoning models has improved the performance of Large Language Models (LLMs) across many tasks which involve logical steps. One linguistic task that could benefit from this framing is idiomaticity detection, as a potentially idiomatic expression must first be understood before it can be disambiguated and serves as a basis for reasoning. In this paper, we explore how reasoning capabilities in LLMs affect idiomaticity detection performance and examine the effect of model size. We evaluate, as open source representative models, the suite of DeepSeek-R1 distillation models ranging from 1.5B to 70B parameters across four idiomaticity detection datasets. We find the effect of reasoning to be smaller and more varied than expected. For smaller models, producing chain-of-thought (CoT) reasoning increases performance from Math-tuned intermediate models, but not to the levels of the base models, whereas larger models (14B, 32B, and 70B) show modest improvements. Our in-depth analyses reveal that larger models demonstrate good understanding of idiomaticity, successfully producing accurate definitions of expressions, while smaller models often fail to output the actual meaning. For this reason, we also experiment with providing definitions in the prompts of smaller models, which we show can improve performance in some cases.

成语识别推理增强大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。