seqBench通过可调参数测试大模型的顺序推理极限,发现其准确率随逻辑深度指数下降。
seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs
- 设计可调节的多维基准,控制逻辑深度、回溯步数和噪声比
- 顶尖模型在逻辑深度超过阈值后准确率呈指数级崩溃
- 适合研究大模型推理缺陷及未来模型评估
我们提出 seqBench,一个参数化基准,用于通过精确、多维度控制关键复杂度维度,探测大语言模型(LLMs)的顺序推理极限。seqBench 可系统性调整:(1)逻辑深度,即解决任务所需的序列操作数量;(2)沿最优路径的回溯步数,量化为满足延迟前提而需重新访问先前状态的频率(如遇到上锁门后才去取钥匙);(3)噪声比,即环境中支持性事实与干扰性事实的比例。对主流 LLMs 的评估显示存在普遍失败模式:准确率在模型特定逻辑深度阈值后呈指数级下降。与现有基准不同,seqBench 的细粒度控制支持对推理失败进行靶向分析,揭示通用缩放规律与统计极限。我们发现,即使顶级模型在结构化推理任务中也系统性失败,尽管搜索复杂度极低,凸显其常识推理能力的关键局限。该基准设计可随模型演进持续更新,数据集已公开,旨在推动对大模型推理能力更深入的科学探究,以明确其真实潜力与当前边界,服务于稳健的现实应用。
原文摘要 · Abstract (English)
We introduce seqBench, a parametrized benchmark for probing sequential reasoning limits in Large Language Models (LLMs) through precise, multi-dimensional control over several key complexity dimensions. seqBench allows systematic variation of (1) the logical depth, defined as the number of sequential actions required to solve the task; (2) the number of backtracking steps along the optimal path, quantifying how often the agent must revisit prior states to satisfy deferred preconditions (e.g., retrieving a key after encountering a locked door); and (3) the noise ratio, defined as the ratio between supporting and distracting facts about the environment. Our evaluations on state-of-the-art LLMs reveal a universal failure pattern: accuracy collapses exponentially beyond a model-specific logical depth. Unlike existing benchmarks, seqBench's fine-grained control facilitates targeted analyses of these reasoning failures, illuminating universal scaling laws and statistical limits, as detailed in this paper alongside its generation methodology and evaluation metrics. We find that even top-performing models systematically fail on seqBench's structured reasoning tasks despite minimal search complexity, underscoring key limitations in their commonsense reasoning capabilities. Designed for future evolution to keep pace with advancing models, the seqBench datasets are publicly released to spur deeper scientific inquiry into LLM reasoning, aiming to establish a clearer understanding of their true potential and current boundaries for robust real-world application.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。