测试大模型在复杂拓扑推理任务中的表现,发现顶级模型仍难解多数难题。
TopoBench: Benchmarking LLMs on Hard Topological Reasoning
- 构建六类拓扑谜题基准,分三难度层级评估模型能力
- 顶尖模型在难题上正确率不足25%,两类谜题几乎无法解决
- 失败主因是空间约束提取与维持困难,而非推理能力不足
解决拓扑网格谜题需要对全局空间不变量(如连通性、环闭合、区域对称性)进行推理,即使最先进的大语言模型也难以应对。为在受控环境下研究此类能力,我们提出TopoBench,一个涵盖六类谜题家族、三个难度级别的基准。我们在该基准上评估强推理型LLM,发现即使是前沿模型,也仅能解决不到四分之一的难题,其中两类谜题几乎无法破解。为探究失败原因是否源于推理局限或空间约束的提取与维持困难,我们标注了750条思维链轨迹,并建立错误分类体系,识别出四种可能的因果失败模式,随后通过针对性干预模拟每种错误类型进行验证。结果显示,过早承诺和约束遗忘等错误模式直接影响解题能力,而重复推理仅为搜索过程的良性副产物。最后,我们考察了提示引导、单元对齐网格表示及工具化约束检查等缓解策略,发现瓶颈在于从空间表征中提取约束,而非在其上进行推理。代码与数据已开源至github.com/mayug/topobench-benchmark。
原文摘要 · Abstract (English)
Solving topological grid puzzles requires reasoning over global spatial invariants such as connectivity, loop closure, and region symmetry and remains challenging for even the most powerful large language models (LLMs). To study these abilities under controlled settings, we introduce TopoBench, a benchmark of six puzzle families across three difficulty levels. We evaluate strong reasoning LLMs on TopoBench and find that even frontier models solve fewer than one quarter of hard instances, with two families nearly unsolved. To investigate whether these failures stem from reasoning limitations or from difficulty extracting and maintaining spatial constraints, we annotate 750 chain of thought traces with an error taxonomy that surfaces four candidate causal failure modes, then test them with targeted interventions simulating each error type. These interventions show that certain error patterns like premature commitment and constraint forgetting have a direct impact on the ability to solve the puzzle, while repeated reasoning is a benign effect of search. Finally we study mitigation strategies including prompt guidance, cell-aligned grid representations and tool-based constraint checking, finding that the bottleneck lies in extracting constraints from spatial representations and not in reasoning over them. Code and data are available at github.com/mayug/topobench-benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。