评测大模型将自然语言转为可执行JQL查询的能力，发现性能差距巨大。
Jackal: A Real-World Execution-Based Benchmark Evaluating Large Language Models on Text-to-JQL Tasks
- 构建10万条真实Jira数据的自然语言到JQL映射基准，含执行结果验证。
- 最佳模型仅60.3%执行准确率，短句和语义相似查询表现极差。
- 适合关注企业级数据查询、大模型实际应用能力的研究者使用。
企业团队依赖Jira查询语言（JQL）从Jira中检索和过滤问题。然而，目前尚无公开、真实世界、基于执行结果的自然语言到JQL映射基准。我们提出Jackal，一个包含10万条自然语言请求与经验证的JQL查询及在超过20万问题的实时Jira实例上执行结果的大型基准。每个JQL查询关联四种用户请求类型：（i）长自然语言，（ii）短自然语言，（iii）语义相似，（iv）语义精确。我们发布Jackal，包含10万条文本到JQL对、执行评分工具包及可复现的Jira实例快照。报告了23个大语言模型（涵盖参数量、开源与闭源）在执行准确率、精确匹配和规范精确匹配上的结果。本文报告的是Jackal-5K（5,000条子集）的结果：最佳模型（Gemini 2.5 Pro）在四类请求上平均执行准确率为60.3%。性能差异显著：长自然语言86.0%，短自然语言35.7%，语义相似22.7%，语义精确99.3%。通过评估大模型生成正确且可执行的JQL能力，Jackal揭示了当前顶级大模型的局限性，并为未来企业数据查询研究设立了新的执行基准。
原文摘要 · Abstract (English)
Enterprise teams rely on the Jira Query Language (JQL) to retrieve and filter issues from Jira. Yet, to our knowledge, there is no open, real-world, execution-based benchmark for mapping natural language queries to JQL. We introduce Jackal, a novel, large-scale text-to-JQL benchmark comprising 100,000 natural language (NL) requests paired with validated JQL queries and execution-based results on a live Jira instance with over 200,000 issues. To reflect real-world usage, each JQL query is associated with four types of user requests: (i) Long NL, (ii) Short NL, (iii) Semantically Similar, and (iv) Semantically Exact. We release Jackal, a corpus of 100,000 text-to-JQL pairs, together with an execution-based scoring toolkit, and a static snapshot of the evaluated Jira instance for reproducibility. We report text-to-JQL results on 23 Large Language Models (LLMs) spanning parameter sizes, open and closed source models, across execution accuracy, exact match, and canonical exact match. In this paper, we report results on Jackal-5K, a 5,000-pair subset of Jackal. On Jackal-5K, the best overall model (Gemini 2.5 Pro) achieves only 60.3% execution accuracy averaged equally across four user request types. Performance varies significantly across user request types: (i) Long NL (86.0%), (ii) Short NL (35.7%), (iii) Semantically Similar (22.7%), and (iv) Semantically Exact (99.3%). By benchmarking LLMs on their ability to produce correct and executable JQL queries, Jackal exposes the limitations of current state-of-the-art LLMs and sets a new, execution-based challenge for future research in Jira enterprise data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。