arXiv:2607.04784cs.SEcs.AI2026-07中稿 · ISSTA 2026

构建动态可控难度的时序推理测试框架,揭示大模型推理缺陷。

A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

论文配图:A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation
图 1 · 摘自论文原文
  • 将时序推理建模为约束满足问题,实现逻辑复杂度精确控制。
  • 发现模型表现与难度指标强负相关(r≈-0.96),验证机制有效性。
  • 暴露推理过程与答案不一致,中等模型虚假猜测率约28%。

定义大推理模型(LRMs)的推理边界并确保其可靠性仍是关键挑战。现有基准多依赖易受污染的静态数据集或缺乏精细难度控制的合成任务,且标准结果评估常因忽略推理过程而掩盖错误。为此,我们提出TRACE框架,通过Allen区间代数将时序推理建模为约束满足问题,实现逻辑复杂度精准调控,并引入基于轨迹的验证判断器以检验推理忠实性。基于该框架,我们构建了包含1,200个合成测试实例的TRACEBench,覆盖不同难度等级。在该基准上评估了八种主流LRMs,结果表明模型性能与我们的难度度量存在强负相关(皮尔逊相关系数约-0.96),验证了难度控制的有效性。此外,轨迹分析揭示推理有效性和最终答案间存在显著差异,中等规模模型的虚假猜测率约为28%。还诊断出尺度相关的失效模式:小型模型表现为退化循环,先进架构则出现推理爆炸。因此,TRACE提供了一个自动化、可靠的平台,用于评估LRMs的真实时序推理能力。

原文摘要 · Abstract (English)

Defining the reasoning boundaries and ensuring the reliability of Large Reasoning Models (LRMs) remains a critical challenge. Current benchmarks primarily rely on static datasets susceptible to data contamination or synthetic tasks lacking fine-grained difficulty control. Furthermore, standard outcome-based evaluations often conceal reasoning flaws by neglecting the reasoning process. To address these limitations, we introduce TRACE, a testing framework that models temporal reasoning as constraint satisfaction problems via Allen's Interval Algebra. This approach enables precise regulation of logical complexity and incorporates a Trace-Based Verification Oracle to validate reasoning faithfulness. Using this framework, we construct TRACEBench, an extensive benchmark comprising 1,200 synthesized test instances across graded difficulty levels. We employ TRACE to evaluate eight widely used LRMs on TRACEBench. The results confirm a strong negative correlation between model performance and our difficulty metric (Pearson's r approximately -0.96), validating the effectiveness of our difficulty control mechanism. Moreover, our trace-based analysis exposes significant discrepancies between reasoning validity and final answers, revealing a high spurious guessing rate of approximately 28% in mid-sized models. In addition, we diagnose scale-dependent failure modes, ranging from Degenerative Loops in small models to Reasoning Explosion in advanced architectures. TRACE thus provides a robust, automated platform for benchmarking the true temporal reasoning capabilities of LRMs.

时序推理大模型评估测试框架逻辑验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。