测试大模型在最基础长链推理任务上的表现，发现推理模型仍有明显短板。
How Well Do LLMs Perform on the Simplest Long-Chain Reasoning Tasks: An Empirical Study on the Equivalence Class Problem
- 用等价类问题检验大模型长链推理能力，覆盖多种变量与连接概率。
- 非推理模型完全失败，推理模型虽更好但仍未彻底解决该问题。
- 不同模型的难点不同：非推理模型在混沌点最差，推理模型在路径最长处最弱。
近年来大语言模型（LLMs）取得了显著进展，但其在推理任务中的表现，尤其是长链推理任务，仍不清晰。本文评估了代表性推理与非推理类大模型在最基础的长链推理任务——等价类问题（ECP）上的表现，即根据一组随机生成的等价关系判断两个变量是否相等。实验涵盖多种问题实例，包括不同变量数、连通概率、提示方式等因素。结果表明，非推理类模型在ECP上完全失效，而推理类模型虽有显著提升但仍无法完全解决该问题。值得注意的是，在固定变量数下，非推理模型最难处理的实例出现在连通概率为ln n/(n-1)的相变点，表明问题具有混沌特性；而推理模型最困难的实例则对应图的最大直径，反映其推理难度随路径增长而上升。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved great improvements in recent years. Nevertheless, it still remains unclear how good LLMs are for reasoning tasks, especially for long-chain ones. In this paper, we evaluate LLMs' performance on the simplest yet long-chain reasoning task, namely the Equivalence Class Problem (ECP), i.e., determining whether two variables are equal given a set of randomly generated equivalence relations. We consider both reasoning and non-reasoning representative LLMs over a large variety of problem instances, ranging over different numbers of variables, connectivity probabilities, prompts, and other factors. The experimental results show that non-reasoning LLMs fail ECP, while reasoning models are significantly better but still struggle to completely solve this problem. Interestingly, considering various connectivity probabilities with a fixed number of variables, we observe that, for non-reasoning models, the hardest problem instances coincide with the phase transition point of ln n/(n-1), suggesting the chaos of the problem; in contrast, for reasoning models, the hardest ones coincide with the biggest diameter, suggesting the reasoning difficulty of the problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。