用百万级难例测试大模型的内存计算极限,发现其推理常冗长重复。
RegexPSPACE: A Benchmark for Evaluating LLM Reasoning on PSPACE-complete Regex Problems
- 构建双指数空间搜索生成百万级正则表达式难题数据集
- 6个大模型在等价性与最小化任务上普遍出现冗余重复错误
- 首个量化评估大模型空间计算能力的基准,适合研究推理瓶颈者
大型语言模型(LLMs)在自然语言处理、数学推理和编程任务中表现优异,近期的大规模推理模型(LRMs)更强调显式推理过程。然而,其计算能力的边界,尤其是受有限上下文窗口限制的空间复杂度,仍不清晰。现有研究多聚焦于NP类问题,本文将边界推至两个PSPACE完全的正则表达式(regex)问题:等价性判定(RegexEQ)与最小化(RegexMin)。这类问题需大规模搜索空间,是更严格的计算能力检验标准。我们通过双指数空间探索,构建了超百万条标注实例的数据集,并采用严格筛选流程建立基准。对6个LLMs和5个LRMs进行广泛评测,发现常见失败模式包括冗长表达与重复输出。该工作首次实证揭示了大模型与推理模型在空间计算上的局限性,提供了评估其高级推理能力的新框架。代码已公开于https://github.com/hyundong98/RegexPSPACE。
原文摘要 · Abstract (English)
Large language models (LLMs) show strong performance across natural language processing (NLP), mathematical reasoning, and programming, and recent large reasoning models (LRMs) further emphasize explicit reasoning. Yet their computational limits, particularly spatial complexity constrained by finite context windows, remain poorly understood. While recent works often focus on problems within the NP complexity class, we push the boundary by introducing a novel benchmark grounded in two PSPACE-complete regular expression (regex) problems: equivalence decision (RegexEQ) and minimization (RegexMin). PSPACE-complete problems serve as a more rigorous standard for assessing computational capacity, as their solutions require massive search space exploration. We perform a double-exponential space exploration to construct a labeled dataset of over a million regex instances with a sound filtering process to build the benchmark. We conduct extensive evaluations on 6 LLMs and 5 LRMs of varying scales, revealing common failure patterns such as verbosity and repetition. With its well-defined structure and quantitative evaluation metrics, this work presents the first empirical investigation into the spatial computational limitations of LLMs and LRMs, offering a new framework for evaluating their advanced reasoning capabilities. Our code is available at https://github.com/hyundong98/RegexPSPACE .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。