测试大模型处理SPARQL查询的能力，发现生成语义正确的查询仍很困难。
Assessing SPARQL capabilities of Large Language Models
- 构建新基准测试框架，量化评估大模型对SPARQL的语法与语义理解能力。
- 顶尖模型可纠正基础语法错误，但复杂查询的语义生成准确率仍低。
- 适合关注知识图谱与大模型融合的研究者或开发者参考。
大语言模型（LLMs）与知识图谱（KGs）的结合为知识驱动应用带来巨大潜力。一种可能的融合方式是解释和生成形式化语言，如语义网中的语言，其中SPARQL是访问知识图谱的核心技术。本文聚焦于量化评估当前大模型在无需微调情况下处理SPARQL的能力，尤其针对SPARQL SELECT查询。我们在LLM-KG-Bench框架中设计了多种自动化评测任务，涵盖语法、语义读取、语义生成以及知识图谱提示的影响等维度，并对多个GPT、Gemini和Claude模型进行了评估。结果表明，处理SPARQL SELECT查询对现有大模型仍具挑战性，且表现高度依赖模型类型与任务复杂度。尽管顶级模型能轻松修复基础语法错误，但在生成语义正确查询方面仍存在显著困难。
原文摘要 · Abstract (English)
The integration of Large Language Models (LLMs) with Knowledge Graphs (KGs) offers significant synergistic potential for knowledge-driven applications. One possible integration is the interpretation and generation of formal languages, such as those used in the Semantic Web, with SPARQL being a core technology for accessing KGs. In this paper, we focus on measuring out-of-the box capabilities of LLMs to work with SPARQL and more specifically with SPARQL SELECT queries applying a quantitative approach. We implemented various benchmarking tasks in the LLM-KG-Bench framework for automated execution and evaluation with several LLMs. The tasks assess capabilities along the dimensions of syntax, semantic read, semantic create, and the role of knowledge graph prompt inclusion. With this new benchmarking tasks, we evaluated a selection of GPT, Gemini, and Claude models. Our findings indicate that working with SPARQL SELECT queries is still challenging for LLMs and heavily depends on the specific LLM as well as the complexity of the task. While fixing basic syntax errors seems to pose no problems for the best of the current LLMs evaluated, creating semantically correct SPARQL SELECT queries is difficult in several cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。