首个评测多路径逻辑推理的基准,揭示大模型易过早定论、忽略其他解法的缺陷。
LogicGraph : Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verification
- 通过神经符号框架生成带多解路径的逻辑题,含深度推理与干扰项
- 实测主流模型在深层推理时解法覆盖率显著下降,暴露探索不足问题
- 适合研究模型可解释性与推理多样性,推动下一代推理系统发展
当前大语言模型评估多聚焦于单一正确证明的收敛式逻辑推理,但真实问题常存在多种有效推导路径。为此,我们提出LogicGraph,首个系统评估多路径逻辑推理的基准,基于神经符号框架构建,利用逆向逻辑生成与语义实例化,产出由高深度多路径推理和内在逻辑干扰构成的可验证问题,每个实例均配有完整的最小证明集。我们还设计无参考评价框架,可严格评估模型在收敛与发散两种情形下的表现。实验显示,现有先进模型普遍过早锁定单一路径,难以探索其他解法,且随着推理深度增加,解法覆盖率差距显著扩大。LogicGraph揭示了这一分化缺口,并为未来改进提供可操作洞察。代码与数据将公开于https://github.com/kkkkarry/LogicGraph。
原文摘要 · Abstract (English)
Evaluations of large language models (LLMs) primarily emphasize convergent logical reasoning, where success is defined by producing a single correct proof. However, many real-world reasoning problems admit multiple valid derivations, requiring models to explore diverse logical paths rather than committing to one route. To address this limitation, we introduce LogicGraph, the first benchmark aimed to systematically evaluate multi-path logical reasoning, constructed via a neuro-symbolic framework that leverages backward logic generation and semantic instantiation. This pipeline yields solver-verified reasoning problems formalized by high-depth multi-path reasoning and inherent logical distractions, where each instance is associated with an exhaustive set of minimal proofs. We further propose a reference-free evaluation framework to rigorously assess model performance in both convergent and divergent regimes. Experiments on state-of-the-art language models reveal a common limitation: models tend to commit early to a single route and fail to explore alternatives, and the coverage gap grows substantially with reasoning depth. LogicGraph exposes this divergence gap and provides actionable insights to motivate future improvements. Our code and data will be released at https://github.com/kkkkarry/LogicGraph.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。