用大模型生成学术知识图谱查询,提升准确率。
FIRESPARQL: A LLM-based Framework for SPARQL Query Generation over Scholarly Knowledge Graphs
- 用微调大模型+检索增强,改进自然语言转SPARQL
- 在SciQA上达0.90的查询准确率和0.85的结果准确率
- 适合需要精准学术问答的科研人员使用
面向学术知识图谱(SKGs)的问答任务因内容复杂和图结构繁复而极具挑战。尽管大语言模型(LLM)可用于将自然语言问题(NLQ)转化为SPARQL查询,但其生成效果受限于对SKG特定内容和底层模式的接触不足。我们识别出两类主要错误:(i) 结构不一致,如三元组缺失或冗余;(ii) 语义错误,即虽结构正确却使用了错误的实体或属性。为此,我们提出FIRESPARQL框架,以微调后的LLM为核心,支持通过检索增强生成(RAG)提供上下文,并引入查询纠错层。我们在SciQA基准上评估多种配置(零样本、零样本+RAG、单样本、微调、微调+RAG),并对比基线与先进方法。使用BLEU和ROUGE衡量查询准确率,采用宽松精确匹配(RelaxedEM)衡量结果准确率。实验表明,微调方案表现最优,在测试集上达到0.90的ROUGE-L查询准确率和0.85的RelaxedEM结果准确率。
原文摘要 · Abstract (English)
Question answering over Scholarly Knowledge Graphs (SKGs) remains a challenging task due to the complexity of scholarly content and the intricate structure of these graphs. Large Language Model (LLM) approaches could be used to translate natural language questions (NLQs) into SPARQL queries; however, these LLM-based approaches struggle with SPARQL query generation due to limited exposure to SKG-specific content and the underlying schema. We identified two main types of errors in the LLM-generated SPARQL queries: (i) structural inconsistencies, such as missing or redundant triples in the queries, and (ii) semantic inaccuracies, where incorrect entities or properties are shown in the queries despite a correct query structure. To address these issues, we propose FIRESPARQL, a modular framework that supports fine-tuned LLMs as a core component, with optional context provided via retrieval-augmented generation (RAG) and a SPARQL query correction layer. We evaluate the framework on the SciQA Benchmark using various configurations (zero-shot, zero-shot with RAG, one-shot, fine-tuning, and fine-tuning with RAG) and compare the performance with baseline and state-of-the-art approaches. We measure query accuracy using BLEU and ROUGE metrics, and query result accuracy using relaxed exact match(RelaxedEM), with respect to the gold standards containing the NLQs, SPARQL queries, and the results of the queries. Experimental results demonstrate that fine-tuning achieves the highest overall performance, reaching 0.90 ROUGE-L for query accuracy and 0.85 RelaxedEM for result accuracy on the test set.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。