测试大模型处理图数据库模糊查询的能力，发现普遍表现不佳。
AmbiGraph-Eval: Can LLMs Effectively Handle Ambiguous Graph Queries?
- 提出三类图查询模糊性分类，覆盖属性、关系及组合场景
- 构建真实世界模糊查询数据集，含专家验证的正确答案
- 9个主流大模型均在模糊查询上表现有限，暴露出关键缺陷
大型语言模型（LLMs）在将自然语言转换为数据库查询方面表现出强大能力，尤其是在处理复杂图结构数据时。然而，现实中的查询常包含固有模糊性，而图结构的相互关联性会放大这些挑战，导致意外或错误的查询结果。为此，我们提出了一种图查询模糊性的分类体系，包括三类主要类型：属性模糊性、关系模糊性和属性-关系模糊性，每类又细分为同实体与跨实体场景。我们引入了AmbiGraph-Eval，一个由真实世界模糊查询及其专家验证的图查询答案组成的新基准。对9个代表性大模型的评估显示，即使顶尖模型在处理模糊图查询时仍面临困难。研究结果揭示了当前模型在模糊性处理上的显著差距，推动未来针对特定解析技术的研究。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have recently demonstrated strong capabilities in translating natural language into database queries, especially when dealing with complex graph-structured data. However, real-world queries often contain inherent ambiguities, and the interconnected nature of graph structures can amplify these challenges, leading to unintended or incorrect query results. To systematically evaluate LLMs on this front, we propose a taxonomy of graph-query ambiguities, comprising three primary types: Attribute Ambiguity, Relationship Ambiguity, and Attribute-Relationship Ambiguity, each subdivided into Same-Entity and Cross-Entity scenarios. We introduce AmbiGraph-Eval, a novel benchmark of real-world ambiguous queries paired with expert-verified graph query answers. Evaluating 9 representative LLMs shows that even top models struggle with ambiguous graph queries. Our findings reveal a critical gap in ambiguity handling and motivate future work on specialized resolution techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。