评测大模型对中文成语的理解与使用能力，发现其语义把握仍有明显短板。
Chengyu-Bench: Benchmarking Large Language Models for Chinese Idiom Understanding and Use
- 构建三类任务：情感判断、用法正误识别、开放式填空。
- 模型在情感判断上准确率超95%，但上下文使用正确率仅约85%。
- 揭示模型多因误解成语本义而误用，适合中文NLP研究者参考。
中文成语是富含历史文化的四字短语，其字面翻译常无法传达真实含义，给语言模型的理解与使用带来挑战。现有基准测试多聚焦单一任务，如选择题填空、孤立翻译或简单改写。本文提出Chengyu-Bench，一个涵盖三类任务的综合性基准：（1）评价语义，判断成语情感倾向；（2）适用性检测，在上下文中识别错误用法；（3）开放填空，在无选项情况下补全长段落。该基准包含2,937个经人工验证的样本，覆盖1,765个常见成语，数据源自多样语料库。我们评估了主流大模型，发现它们在评价语义任务上准确率超过95%，但在适用性任务上仅约85%，开放填空任务中顶级模型的单次预测准确率仅为约40%。错误分析表明，多数失误源于对成语本义的根本误解。结果表明，尽管模型能较好判断成语情感，但在理解文化与语境细微差别方面仍存在显著不足。基准与源代码已公开于：https://github.com/sofyc/ChengyuBench。
原文摘要 · Abstract (English)
Chinese idioms (Chengyu) are concise four-character expressions steeped in history and culture, whose literal translations often fail to capture their full meaning. This complexity makes them challenging for language models to interpret and use correctly. Existing benchmarks focus on narrow tasks - multiple-choice cloze tests, isolated translation, or simple paraphrasing. We introduce Chengyu-Bench, a comprehensive benchmark featuring three tasks: (1) Evaluative Connotation, classifying idioms as positive or negative; (2) Appropriateness, detecting incorrect idiom usage in context; and (3) Open Cloze, filling blanks in longer passages without options. Chengyu-Bench comprises 2,937 human-verified examples covering 1,765 common idioms sourced from diverse corpora. We evaluate leading LLMs and find they achieve over 95% accuracy on Evaluative Connotation, but only ~85% on Appropriateness and ~40% top-1 accuracy on Open Cloze. Error analysis reveals that most mistakes arise from fundamental misunderstandings of idiom meanings. Chengyu-Bench demonstrates that while LLMs can reliably gauge idiom sentiment, they still struggle to grasp the cultural and contextual nuances essential for proper usage. The benchmark and source code are available at: https://github.com/sofyc/ChengyuBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。