LLMs解语言谜题能力差，因难以处理复杂形态语言。
UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?
- 用语言学特征标注629道低资源语言谜题，定位模型弱点。
- 形态越复杂，模型表现越差；与英语共有的特征更易解决。
- 分词为词素能提升解题率，提示需更智能的分词器。
大型语言模型（LLMs）在推理任务中表现潜力，但在语言学谜题上始终表现不佳。这些谜题源自语言学奥林匹克竞赛，为评估低资源语言中的语言推理能力提供了极少干扰的环境。本研究分析了41种低资源语言中的629个问题，通过语言学启发的特征标注揭示模型缺陷。结果显示，模型在涉及高形态复杂性的谜题上表现较差，而在包含英语中也存在的语言特征的谜题上表现更好。此外，将单词拆分为词素作为预处理可显著提高可解性，表明需要更具备语言特异性、更智能的分词器。这些发现为低资源语言的语言推理与建模挑战提供了重要洞见。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated potential in reasoning tasks, but their performance on linguistics puzzles remains consistently poor. These puzzles, often derived from Linguistics Olympiad (LO) contests, provide a minimal contamination environment to assess LLMs' linguistic reasoning abilities across low-resource languages. This work analyses LLMs' performance on 629 problems across 41 low-resource languages by labelling each with linguistically informed features to unveil weaknesses. Our analyses show that LLMs struggle with puzzles involving higher morphological complexity and perform better on puzzles involving linguistic features that are also found in English. We also show that splitting words into morphemes as a pre-processing step improves solvability, indicating a need for more informed and language-specific tokenisers. These findings thus offer insights into some challenges in linguistic reasoning and modelling of low-resource languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。