用轻量元数据实现自然语言到SPARQL的实时生成,兼顾准确与高效。
SPARQL-LLM: Real-Time SPARQL Query Generation from Natural Language Questions
- 基于轻量元数据构建三组件架构,分步完成索引、提示和查询生成。
- 在多语言挑战中F1提升24%,支持复杂跨源生物信息查询。
- 比同类系统快36倍,单次查询成本不足0.01美元,适合实时部署。
大语言模型推动了从自然语言生成结构化查询(如SPARQL)的新方法,但多数研究仅关注单一数据源的准确率,忽视联邦查询能力、运行时效率及生成成本。为此,本文提出SPARQL-LLM——一个开源、无关三元组存储的轻量级元数据驱动方法,可从自然语言生成SPARQL查询。其架构包含元数据索引、提示构建与查询生成执行三个模块。我们在一项前沿多语言挑战及生物信息学领域三大知识图谱的问答数据集上进行系统评估。结果表明,该方法在挑战任务中F1分数提升24%,支持英语、西班牙语等高资源语言,并能生成复杂且跨源的生物信息查询。此外,其处理速度较参与挑战的其他系统快达36倍,单次查询成本最高仅0.01美元,适用于低延迟、低成本的实时文本转SPARQL应用。实际部署案例可见于基于真实去中心化知识图谱的聊天系统:https://www.expasy.org/chat。
原文摘要 · Abstract (English)
The advent of large language models is contributing to the emergence of novel approaches that promise to better tackle the challenge of generating structured queries, such as SPARQL queries, from natural language. However, these new approaches mostly focus on response accuracy over a single source while ignoring other evaluation criteria, such as federated query capability over distributed data stores, as well as runtime and cost to generate SPARQL queries. Consequently, they are often not production-ready or easy to deploy over (potentially federated) knowledge graphs with good accuracy. To mitigate these issues, in this paper, we extend our previous work and describe and systematically evaluate SPARQL-LLM, an open-source and triplestore-agnostic approach, powered by lightweight metadata, that generates SPARQL queries from natural language text. First, we describe its architecture, which consists of dedicated components for metadata indexing, prompt building, and query generation and execution. Then, we evaluate it based on a state-of-the-art challenge with multilingual questions, and a collection of questions from three of the most prevalent knowledge graphs within the field of bioinformatics. Our results demonstrate a substantial increase of 24% in the F1 Score on the state-of-the-art challenge, adaptability to high-resource languages such as English and Spanish, as well as ability to form complex and federated bioinformatics queries. Furthermore, we show that SPARQL-LLM is up to 36x faster than other systems participating in the challenge, while costing a maximum of $0.01 per question, making it suitable for real-time, low-cost text-to-SPARQL applications. One such application deployed over real-world decentralized knowledge graphs can be found at https://www.expasy.org/chat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。