探究大模型解谜题困难的原因，发现其表现远低于人类。
What Makes Cryptic Crosswords Challenging for LLMs?
- 对比Gemma2、LLaMA3和ChatGPT在密语填字谜上的表现
- 模型准确率显著低于人类水平，存在明显差距
- 适合研究语言理解与推理的局限性
密语填字谜依赖常识和多层级语言操控能力，涉及多种文字游戏。先前研究显示，即使现代自然语言模型（包括大语言模型，LLMs）也难以解决此类谜题。然而，关于其表现不佳的具体原因尚无充分研究。本文建立了Gemma2、LLaMA3和ChatGPT三个主流LLM在该任务上的基准表现，结果表明它们的准确率仍显著低于人类。同时，我们深入分析了模型表现受限的根本原因。相关代码与数据集已公开于https://github.com/bodasadallah/decrypting-crosswords。
原文摘要 · Abstract (English)
Cryptic crosswords are puzzles that rely on general knowledge and the solver's ability to manipulate language on different levels, dealing with various types of wordplay. Previous research suggests that solving such puzzles is challenging even for modern NLP models, including Large Language Models (LLMs). However, there is little to no research on the reasons for their poor performance on this task. In this paper, we establish the benchmark results for three popular LLMs: Gemma2, LLaMA3 and ChatGPT, showing that their performance on this task is still significantly below that of humans. We also investigate why these models struggle to achieve superior performance. We release our code and introduced datasets at https://github.com/bodasadallah/decrypting-crosswords.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。