提出图结构框架,评估大模型推理的可信度与鲁棒性。
Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework

- 构建图模型捕捉推理过程的语义与结构一致性
- 新指标GRCS能有效识别幻觉和错误推理模式
- 改进解码策略使推理更可信,适合关注模型可靠性研究者
大语言模型在推理中常出现逻辑错误且不可靠,而传统自一致性(SC)仅检查最终答案一致,忽视中间步骤的有效性。本文提出GRAPHEVAL图结构框架,将不确定性量化(UQ)视为整体推理可信度问题。引入新指标图推理一致性得分(GRCS),衡量推理空间中的语义-结构共识,可捕捉病理模式崩溃与自信幻觉。实验发现,GRCS是唯一在大模型与小模型中均与推理可信度显著负相关的指标。进一步提出图自一致性(GSC)解码策略,基于中位数路径选择,以牺牲轻微准确率为代价提升推理可信度,在小模型中暴露了传统SC因侥幸猜对导致的虚高,在大模型中保持或提升准确率。通过对抗性中位数消融实验,验证了GSC选择路径为关键“承重路径”,强行偏离会降低推理可信度,甚至导致准确率下降。
原文摘要 · Abstract (English)
Large-Language Models (LLMs) can be prone to flawed and unfaithful reasoning that decoding strategies like Self-Consistency (SC) fail to detect as they evaluate only final-answer agreement while ignoring the logical validity of intermediate steps. This raises three fundamental questions: How can we reliably quantify uncertainty in LLM reasoning? Can semantic, structural, and causal awareness select more faithful reasoning compared to naïve majority voting? and How robust is reasoning topology under adversarial conditions? To address these questions, we introduce GRAPHEVAL, a graph-based reasoning framework that re-frames uncertainty quantification (UQ) as a holistic reasoning fidelity problem. We propose a novel UQ metric, Graph Reasoning Coherence Score (GRCS), that quantifies semantic-structural consensus of the reasoning space and captures pathological mode collapse and confident hallucinations. We find that GRCS is the only metric that is consistently negatively correlated with reasoning faithfulness across both more capable and smaller models. Additionally, we introduce Graph Self-Consistency (GSC), a medoid-based decoding strategy that trades nominal accuracy for reasoning fidelity, exposing the degree to which SC is inflated by unfaithful lucky guesses in smaller models, while preserving or improving accuracy in more capable ones. Finally, through adversarial medoid ablation, we demonstrate that the GSC-selected path acts as a "load-bearing path" and forcing models away from it degrades reasoning faithfulness and, in targeted cases, causes drops in accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。