为使用多源工具的AI代理设计了源信息敏感的事实核查系统。
ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents
- 根据工具调用记录分解答案为原子命题,逐个验证来源归属
- 在医疗领域数据上实现0.858的源识别准确率,优于忽略来源的基准
- 能检测并修复错误归因,适合需要高可信度的医疗等关键场景
使用工具的大型语言模型代理越来越多地依赖模型上下文协议(MCP)从异构证据源(如搜索、API、数据库、临床记录和药品目录工具)中获取信息。标准事实性评估通常只检查答案是否被合并证据支持,忽略了关键的溯源敏感性问题:某个说法可能在某处有依据,却被错误归因。我们称此为跨源混淆。为此提出ProvenanceGuard,一种面向MCP驱动答案的源感知验证器。它接收包含稳定工具ID、源ID和原始输出的MCP追踪日志;将答案分解为原子命题;将每个命题路由至对应来源证据;通过自然语言推断(NLI)与词元对齐代理检查支持性;比对陈述归因与实际路由源;返回逐命题判别结果及整体允许/禁止决策。被阻止的答案可通过检索增强的重写进行修复并重新验证。在281条医疗领域MCP代理轨迹上评估,其中266条经人工标注,共生成2,325个由LLM辅助标记的命题;另有361个命题由人类验证。在40条保留测试集中,ProvenanceGuard实现阻断F1 0.802,源识别准确率0.858,超过不输出命题-源映射的盲基线。在更难的多源基准上,阻断F1达0.846,但源+关系准确率降至0.229,表明语义相近来源间精确归属仍具挑战。修复-再验证可解决全部被阻断答案,常通过保守回退策略完成。在50个受控临床混淆探测中,成功检出所有注入的归因替换,且无残留错误归因。结果表明,源归因是MCP型代理事实性验证中的独立维度。
原文摘要 · Abstract (English)
Tool-using LLM agents increasingly use the Model Context Protocol (MCP) to answer from heterogeneous evidence sources, including search, APIs, databases, clinical records, and formulary tools. Standard factuality metrics usually test whether an answer is supported by pooled evidence, missing a provenance-sensitive failure mode: a claim may be supported somewhere while being attributed to the wrong source. We call this cross-source conflation. We introduce ProvenanceGuard, a source-aware verifier for MCP-grounded answers. It consumes captured MCP traces with stable tool IDs, source IDs, and raw outputs; decomposes answers into atomic claims; routes claims to source-specific evidence; checks support with NLI and a token-alignment proxy; compares stated attribution with the routed source; and returns per-claim verdicts plus an answer-level allow/block decision. Blocked answers can be repaired with retrieval-augmented answer revision and re-verified. We evaluate on 281 medical-domain MCP-agent traces. A 266-trace adjudicated subset yields 2,325 LLM-assisted claim labels split by trace; 361 held-out labels are human-verified. On the 40-trace held-out split, ProvenanceGuard achieves block F1 0.802 and source accuracy 0.858 over 260 source-eligible claims, outperforming source-blind baselines that do not emit claim-to-source IDs. On a harder multi-source benchmark it reaches block F1 0.846, while source-plus-relation accuracy drops to 0.229, showing that exact source ownership remains difficult with semantically close sources. Repair-and-reverify resolves all blocked answers in the full trace set, often via conservative fallback. In 50 controlled clinical conflation probes, ProvenanceGuard detects all injected attribution swaps with no retained wrong attribution. These results show that source attribution is an independent axis for factuality verification in MCP-based agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。