评估大模型生成答案的逻辑完整性，发现高准确率下仍存推理漏洞。
Beyond Factual Accuracy: Evaluating Global Reasoning Integrity in RAG Systems with LogicScore
- 用后向验证机制全局检验推理的完整、必要和确定性。
- 顶尖模型事实准确率达92.85%，但逻辑必要性仅35.11%。
- 适合关注模型推理质量的研究者与开发者使用。
当前检索增强生成（RAG）系统的评估方法存在事实盲区：过度强调单个事实准确性，却忽视长文本回答中的全局逻辑一致性。这导致模型强行拼接信息，生成看似正确却逻辑断裂、存在遗漏或冗余的答案。为此，本文提出 extsc{LogicScore}，从局部事实核查转向全局推理审视。基于命题逻辑规则，该方法通过后向验证机制系统评估三个关键维度：完备性（逻辑推导是否完整）、必要性（是否存在冗余前提）和确定性（答案是否一致可推）。在三组多跳问答数据集（HotpotQA、MusiQue、2WikiMultiHopQA）及超过20种大模型（包括GPT-5、Gemini-3-Pro、LLaMA3及任务微调模型）上的实验表明，主流模型虽具备高事实准确率（如Gemini-3-Pro达92.85%），但在全局推理质量上仍有显著短板（如其必要性仅为35.11%）。本工作建立了一套可靠的逻辑评估标准，强调在大模型发展中需同步重视推理连贯性与事实准确性。
原文摘要 · Abstract (English)
Current evaluation methods for Retrieval Augmented Generation (RAG) suffer from \textit{factual myopia}: they relentlessly emphasize factual accuracy yet neglect global logical integrity in long-form answer generation. This drives models to force unnatural connections, producing factually grounded yet logically incoherent responses with unaddressed gaps, ambiguous links, or redundant premises. To mitigate this, we present \textsc{LogicScore}, shifting from local, fact-by-fact assessment to rigorous global reasoning scrutiny. Grounded in Horn Rules, our approach integrates a backward verification mechanism to systematically evaluate three key reasoning dimensions: \textit{Completeness} (logically sound deduction), \textit{Essentiality} (non-redundancy), and \textit{Determinateness} (consistent answer entailment). Extensive experiments across three multi-hop QA datasets (HotpotQA, MusiQue, and 2WikiMultiHopQA) and over 20 LLMs (including GPT-5, Gemini-3-Pro, LLaMA3, and task-specific tuned models) reveal a critical capability gap: leading models often achieve high factual accuracy (e.g., 92.85\% precision for Gemini-3 Pro) but struggle with global reasoning quality (e.g., 35.11\% Essentiality for Gemini-3 Pro). Our work establishes a robust standard for logical evaluation, highlighting the need to prioritize reasoning coherence alongside factual grounding in LLM development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。