arXiv:2608.00711cs.AI2026-08

首个基于知识拓扑的科学智能体幻觉评估框架,揭示错误如何通过关键节点传播。

Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations

论文配图:Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations
图 1 · 摘自论文原文
  • 构建科学概念图谱,从文献证据中自动提取知识关系。
  • 发现幻觉集中于少数高连接度知识节点,且正确答案常来自错误推理路径。
  • 适合关注科学推理可靠性与模型可解释性的研究人员使用。

大型语言模型(LLM)代理在科学研究中的应用日益广泛,而可靠性至关重要,因为底层知识高度互联。在此背景下,幻觉尤为危险:一个基础概念上的错误声明可能通过多步推理传播,导致整个推理轨迹被污染。现有幻觉基准大多停留在表面层次,将事实孤立看待,并依赖忽略拓扑结构的统一准确率指标。为此,我们提出SCHEMA——首个基于证据、面向拓扑结构的科学代理幻觉评估框架。SCHEMA可自动从基准种子和文献证据中构建科学概念图谱,合成涵盖命题验证、多跳推理、开放式解释和实验代码生成的图谱驱动任务,并采用两种互补诊断方法进行评估。轨迹幻觉分析管道通过拓扑加权严重度分数,在大规模上审计中间推理过程;多代理反事实归因模块则定位特定失败的因果机制。结果表明,幻觉集中在少数高度连接的知识枢纽上,且最终答案准确率与推理路径真实性解耦:模型常通过结构性缺陷的推理得出正确结论。这说明,在高风险科学应用中,仅靠终端准确率无法充分反映代理可靠性,需基于知识拓扑开展机制级评估。代码已开源:https://github.com/circles-post/SCHEMA。

原文摘要 · Abstract (English)

Large language model (LLM) agents are increasingly deployed in scientific research, where reliability is critical and the underlying knowledge is densely interconnected. In such settings, hallucinations are particularly damaging: a single erroneous claim on a foundational concept can propagate through multi-step reasoning and corrupt entire trajectories. Existing hallucination benchmarks largely operate at the surface level, treating facts in isolation and relying on uniform accuracy metrics that ignore this topological structure. We address this gap with SCHEMA, the first evidence-grounded, topology-aware evaluation framework for hallucinations in scientific agents. SCHEMA automatically constructs scientific concept graphs from benchmark seeds and literature evidence, synthesizes graph-grounded tasks spanning claim verification, multi-hop reasoning, open-ended explanation, and experimental code generation, and evaluates agents with two complementary diagnostics. A trajectory hallucination pipeline audits intermediate reasoning at scale via a topology-weighted severity score, while a multi-agent counterfactual attribution module pinpoints the causal mechanism behind selected failures. SCHEMA reveals that hallucinations concentrate at a small set of highly connected knowledge hubs, and that final-answer accuracy decouples from trajectory honesty; models often reach correct conclusions through structurally flawed reasoning. These results indicate that for high-stakes scientific applications, terminal accuracy alone is an insufficient signal of agent reliability, motivating mechanism-level evaluation grounded in knowledge topology. Code is available at https://github.com/circles-post/SCHEMA.

幻觉检测科学推理知识图谱评估框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。