测试大模型解语言谜题能力，发现提示技巧能显著提升推理与翻译表现。
Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
- 用输入输出、思维链和单独表演等提示方法激发模型推理能力。
- 在语言奥赛数据集上，GPT-4 0603 在思维链提示下准确率显著更高。
- 适合研究大模型认知机制与NLP任务优化的学者参考。
本文研究大语言模型（LLMs）在解决复杂语言谜题中的应用，这类任务需高级推理与精准翻译能力，类似于人类认知过程。我们探讨了多种提示技术以增强模型的推理能力并揭示其决策路径，重点关注输入输出提示（IO）、思维链提示（CoT）和单独表演提示（SPP）。基于来自Puzzling Machine竞赛与各类语言奥林匹克竞赛的数据集，采用多维度指标评估GPT-4 0603在不同提示方法下的表现。结果表明，大语言模型在语言推理与复杂翻译任务中具有潜力，同时识别出其在处理语言谜题时的局限性。本研究为自然语言处理（NLP）领域提供了关于优化大模型应用以提升推理与翻译准确性的关键洞见，推动了该领域的持续发展。
原文摘要 · Abstract (English)
This paper investigates the utilization of Large Language Models (LLMs) for solving complex linguistic puzzles, a domain requiring advanced reasoning and adept translation capabilities akin to human cognitive processes. We explore specific prompting techniques designed to enhance ability of LLMs to reason and elucidate their decision-making pathways, with a focus on Input-Output Prompting (IO), Chain-of-Thought Prompting (CoT), and Solo Performance Prompting (SPP). Utilizing datasets from the Puzzling Machine Competition and various Linguistics Olympiads, we employ a comprehensive set of metrics to assess the performance of GPT-4 0603, a prominent LLM, across these prompting methods. Our findings illuminate the potential of LLMs in linguistic reasoning and complex translation tasks, highlighting their capabilities and identifying limitations in the context of linguistic puzzles. This research contributes significantly to the broader field of Natural Language Processing (NLP) by providing insights into the optimization of LLM applications for improved reasoning and translation accuracy, thereby enriching the ongoing dialogue in NLP advancements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。