评测大模型在多种知识源上的推理能力,发现其结构化推理短板明显。
OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases
- 构建多模态知识基准,涵盖文本、知识图谱、代码和逻辑四种类型
- 最强模型在难题集上仅32.2%准确率,结构越复杂准确率越低
- 长推理链不等于好结果,需根据任务复杂度动态调整推理深度
大型语言模型(LLMs)在非结构化文本推理任务中表现突出,但在需要整合结构化外部知识(如知识图谱、代码片段或形式逻辑)时能力显著下降。这一局限部分源于缺乏系统评估模型在多样化结构化知识模态下表现的基准。为此,我们提出 extbf{ extsc{OneEval}},一个全面的基准,专门评估LLMs在四种结构化知识模态(非结构化文本、知识图谱、代码、形式逻辑)及五个关键领域(通用知识、政府、科学、法律、编程)中的知识密集型推理能力。 extsc{OneEval}包含4,019个精心设计的样本,并设立挑战子集 extsc{OneEval} extsubscript{Hard},含1,285个高难度案例。对18个前沿开源与专有大模型的广泛评估揭示三个核心发现:a) 结构化推理存在持续性局限,最强模型在 extsc{OneEval} extsubscript{Hard}上准确率仅为32.2%;b) 随着知识库结构复杂度上升,性能持续下降,从文本推理的53%骤降至形式逻辑的25%;c) 推理链越长收益越小,表明模型需根据任务复杂度智能调节推理深度。我们公开发布 extsc{OneEval}数据集、评估脚本和基线结果,并提供排行榜,以推动结构化知识推理研究进展。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated substantial progress on reasoning tasks involving unstructured text, yet their capabilities significantly deteriorate when reasoning requires integrating structured external knowledge such as knowledge graphs, code snippets, or formal logic. This limitation is partly due to the absence of benchmarks capable of systematically evaluating LLM performance across diverse structured knowledge modalities. To address this gap, we introduce \textbf{\textsc{OneEval}}, a comprehensive benchmark explicitly designed to assess the knowledge-intensive reasoning capabilities of LLMs across four structured knowledge modalities, unstructured text, knowledge graphs, code, and formal logic, and five critical domains (general knowledge, government, science, law, and programming). \textsc{OneEval} comprises 4,019 carefully curated instances and includes a challenging subset, \textsc{OneEval}\textsubscript{Hard}, consisting of 1,285 particularly difficult cases. Through extensive evaluation of 18 state-of-the-art open-source and proprietary LLMs, we establish three core findings: a) \emph{persistent limitations in structured reasoning}, with even the strongest model achieving only 32.2\% accuracy on \textsc{OneEval}\textsubscript{Hard}; b) \emph{performance consistently declines as the structural complexity of the knowledge base increases}, with accuracy dropping sharply from 53\% (textual reasoning) to 25\% (formal logic); and c) \emph{diminishing returns from extended reasoning chains}, highlighting the critical need for models to adapt reasoning depth appropriately to task complexity. We release the \textsc{OneEval} datasets, evaluation scripts, and baseline results publicly, accompanied by a leaderboard to facilitate ongoing advancements in structured knowledge reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。