用数据结构评测大模型的结构性推理能力，发现顶尖模型仍有严重短板。
Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures
- 以20种数据结构为视角，设计可自动构建与评估的推理基准
- 顶级模型在难题上准确率仅0.46/1，整体表现远低于人类水平
- 暴露模型在空间信息和代码自推理上的薄弱，适合算法研究者参考
大语言模型被应用于越来越复杂的多步决策任务，理解其算法推理能力至关重要。然而，现有缺乏诊断性基准。本文提出以数据结构为分析视角：作为算法的基本构件，它们天然探测结构性推理——即对顺序、层次、连通性等关系的理解与操作能力。我们构建了DSR-Bench（数据结构推理基准），涵盖20种数据结构、35种操作和4,140个问题实例。该基准具有层级化任务组织、全自动化生成与评估、细粒度诊断特性。对13个主流LLM的评估显示：最先进模型在挑战性任务上准确率仅为0.46/1。三个辅助探针揭示更深层缺陷：模型在空间数据和上下文丰富场景中表现不佳，且难以推理自身代码。
原文摘要 · Abstract (English)
Large language models (LLMs) are deployed on increasingly complex tasks that require multi-step decision-making. Understanding their algorithmic reasoning abilities is therefore crucial. However, we lack a diagnostic benchmark for evaluating these capabilities. We propose to use data structures as a principled lens: as fundamental building blocks of algorithms, they naturally probe structural reasoning - the ability to understand and manipulate relationships such as order, hierarchy, and connectivity that underpin algorithmic reasoning. We introduce DSR-Bench (Data Structure Reasoning Benchmark), spanning 20 data structures, 35 operations, and 4,140 problem instances. DSR-Bench features hierarchical task organization, fully automated generation and evaluation, and fine-grained diagnostics. Evaluating 13 state-of-the-art LLMs reveals critical limitations: the top-performing model achieves only 0.46/1 on challenging instances. Three auxiliary probes targeting more realistic usages expose further weaknesses: models perform poorly on spatial data and context-rich scenarios, and they struggle to reason over their own code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。