测试大模型对五种复杂形态语言的掌握程度,发现英文强模型在非英语形态上表现差。
IMPACT: Inflectional Morphology Probes Across Complex Typologies
- 构建合成数据集评估大模型对词形变化的理解能力
- 八款多语言模型在非英语形态上普遍表现不佳,尤其识别错误句式时
- 适合研究多语言模型语言理解能力或跨语言语言学评测的学者
大型语言模型(LLMs)在多种多语言基准上表现出显著进展,并被越来越多地用于非英语文本的生成与评估。然而,尽管它们能生成流畅输出,其对目标语言深层语言复杂性的真正理解仍不明确,尤其是在形态学方面。为此,我们提出IMPACT——一个专注于词形变化的合成评估框架,公开发布,用于评估八款多语言模型在阿拉伯语、俄语、芬兰语、土耳其语和希伯来语这五种形态丰富的语言上的表现。IMPACT包含单元测试风格的案例,涵盖共通及语言特有现象,从基本动词变位(如时态、数、性)到阿拉伯语的逆向性别一致、芬兰语与土耳其语的元音和谐等独特特征。评估显示,尽管这些模型在英语上表现良好,但在其他语言及罕见形态模式下表现欠佳,尤其在判断病句时。我们还发现,思维链(Chain of Thought)和思考型模型可能降低性能。本工作揭示了大模型在处理语言复杂性方面的明显不足,指明改进空间。为支持后续研究,我们公开发布IMPACT框架。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown significant progress on various multilingual benchmarks and are increasingly used to generate and evaluate text in non-English languages. However, while they may produce fluent outputs, it remains unclear to what extent these models truly grasp the underlying linguistic complexity of those languages, particularly in morphology. To investigate this, we introduce IMPACT, a synthetically generated evaluation framework focused on inflectional morphology, which we publicly release, designed to evaluate LLM performance across five morphologically rich languages: Arabic, Russian, Finnish, Turkish, and Hebrew. IMPACT includes unit-test-style cases covering both shared and language-specific phenomena, from basic verb inflections (e.g., tense, number, gender) to unique features like Arabic's reverse gender agreement and vowel harmony in Finnish and Turkish. We assess eight multilingual LLMs that, despite strong English performance, struggle with other languages and uncommon morphological patterns, especially when judging ungrammatical examples. We also show that Chain of Thought and Thinking Models can degrade performance. Our work exposes gaps in LLMs' handling of linguistic complexity, pointing to clear room for improvement. To support further research, we publicly release the IMPACT framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。