用编程模拟测试大模型高阶推理能力，高效且有效。
Code Simulation as a Proxy for High-order Tasks in Large Language Models
- 以代码结构模拟自然推理任务的逻辑步骤。
- 强模型虽能执行但易受记忆干扰，依赖模式识别。
- 合成数据可大规模生成，适合评估模型推理韧性。
许多推理、规划和问题解决任务具有内在的算法特性：正确模拟每一步即可正确求解。我们收集了自然任务与合成任务对，评估大语言模型（LLM）的能力。尽管自然任务常需人工精心设计，但研究表明合成数据在多数情况下是更易规模化获取的良好代理。我们采用编程中的常见结构作为自然推理任务构建块的对应物，如线性程序、含关键路径的代码、近似与冗余指令。进一步通过排序问题及嵌套循环测试模型执行能力。合成数据揭示：最强大的LLM虽具备较强执行能力，但过程脆弱，受记忆影响显著，高度依赖模式识别。本研究贡献在于提出一种可扩展的合成测试方法，作为人工标注任务的补充，用于评估LLM的推理能力。
原文摘要 · Abstract (English)
Many reasoning, planning, and problem-solving tasks share an intrinsic algorithmic nature: correctly simulating each step is a sufficient condition to solve them correctly. We collect pairs of naturalistic and synthetic reasoning tasks to assess the capabilities of Large Language Models (LLM). While naturalistic tasks often require careful human handcrafting, we show that synthetic data is, in many cases, a good proxy that is much easier to collect at scale. We leverage common constructs in programming as the counterpart of the building blocks of naturalistic reasoning tasks, such as straight-line programs, code that contains critical paths, and approximate and redundant instructions. We further assess the capabilities of LLMs on sorting problems and repeated operations via sorting algorithms and nested loops. Our synthetic datasets further reveal that while the most powerful LLMs exhibit relatively strong execution capabilities, the process is fragile: it is negatively affected by memorisation and seems to rely heavily on pattern recognition. Our contribution builds upon synthetically testing the reasoning capabilities of LLMs as a scalable complement to handcrafted human-annotated problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。