评测主流大模型生成图数据库查询的能力，发现Claude表现最佳。
Towards Evaluating Large Language Models for Graph Query Generation
- 用少样本提示+RAG+思维链增强大模型生成Cypher语句。
- Claude Sonnet 3.5在准确率上优于GPT-4o、Gemini Pro等模型。
- 适合关注图数据库自动化查询的研究者与开发者参考。
大型语言模型（LLMs）正在重塑生成式人工智能的格局，但应用于图数据库和知识图谱的查询生成时仍面临挑战。尽管已有针对结构化查询语言（SQL）的LLM驱动查询研究，但图数据库领域的相关系统仍不成熟。本文针对使用开放获取的LLMs生成强大的图数据库查询语言Cypher的问题，开展对比研究。我们通过设计的少样本提示及基于思维链（CoT）推理的检索增强生成（RAG）方法，严格评估了多个LLM代理（OpenAI ChatGPT 4o、Claude Sonnet 3.5、Google Gemini Pro 1.5，以及本地部署的Llama 3.1 8B）。实证分析显示，Claude Sonnet 3.5在查询生成准确性方面优于其他模型。此外，我们指出了未来研究方向，以应对现有局限，推动图数据库中LLM驱动查询生成的发展。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are revolutionizing the landscape of Generative Artificial Intelligence (GenAI), with innovative LLM-backed solutions emerging rapidly. However, when applied to database technologies, specifically query generation for graph databases and Knowledge Graphs (KGs), LLMs still face significant challenges. While research on LLM-driven query generation for Structured Query Language (SQL) exists, similar systems for graph databases remain underdeveloped. This paper presents a comparative study addressing the challenge of generating Cypher queries a powerful language for interacting with graph databases using open-access LLMs. We rigorously evaluate several LLM agents (OpenAI ChatGPT 4o, Claude Sonnet 3.5, Google Gemini Pro 1.5, and a locally deployed Llama 3.1 8B) using a designed few-shot learning prompt and Retrieval Augmented Generation (RAG) backed by Chain-of-Thoughts (CoT) reasoning. Our empirical analysis of query generation accuracy reveals that Claude Sonnet 3.5 outperforms its counterparts in this specific domain. Further, we highlight promising future research directions to address the identified limitations and advance LLM-driven query generation for graph databases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。