让大模型实时执行并理解自然语言转Jira查询，提升准确率。
Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL
- 用工具增强大模型，实时调用Jira接口执行查询
- 在10万条真实数据上测试，准确率最高提升9%
- 适合需要精准管理工单的开发团队使用
将自然语言转换为Jira查询语言（JQL）需解决字段引用模糊、实例特定分类值及复杂布尔谓词问题。单次运行的大模型无法发现某Jira实例中实际存在的分类值，也无法验证生成查询是否符合实时数据源，导致对改写或模糊请求的准确率受限。目前缺乏公开的、基于执行的自然语言到JQL映射基准。我们提出Jackal，首个大规模、基于执行的文本到JQL基准，包含10万条经验证的NL-JQL配对，基于超过20万条工单的实时Jira实例。为建立基线，我们提出Agentic Jackal，一种通过Jira MCP服务器和JiraAnchor（基于嵌入相似性搜索的语义检索工具）赋予大模型实时查询执行能力的工具增强型智能体。在9个前沿大模型中，单次运行模型在短自然语言查询上的平均执行准确率为43.4%，表明文本到JQL仍是开放挑战。代理方法使其中7个模型性能提升，最复杂的语言变体相对提升9.0%；在控制消融实验中，仅使用JiraAnchor，分类值准确率从48.7%提升至71.7%，组件字段准确率从16.9%跃升至66.2%。分析指出，固有的语义模糊性（如工单类型歧义和文本字段选择）是主要失败模式，而非值解析错误，为未来工作提供明确方向。我们公开发布基准数据集、所有智能体对话记录及评估代码以支持可复现性。
原文摘要 · Abstract (English)
Translating natural language into Jira Query Language (JQL) requires resolving ambiguous field references, instance-specific categorical values, and complex Boolean predicates. Single-pass LLMs cannot discover which categorical values (e.g., component names or fix versions) actually exist in a given Jira instance, nor can they verify generated queries against a live data source, limiting accuracy on paraphrased or ambiguous requests. No open, execution-based benchmark exists for mapping natural language to JQL. We introduce Jackal, the first large-scale, execution-based text-to-JQL benchmark comprising 100,000 validated NL-JQL pairs on a live Jira instance with over 200,000 issues. To establish baselines on Jackal, we propose Agentic Jackal, a tool-augmented agent that equips LLMs with live query execution via the Jira MCP server and JiraAnchor, a semantic retrieval tool that resolves natural-language mentions of categorical values through embedding-based similarity search. Among 9 frontier LLMs evaluated, single-pass models average only 43.4% execution accuracy on short natural-language queries, highlighting that text-to-JQL remains an open challenge. The agentic approach improves 7 of 9 models, with a 9.0% relative gain on the most linguistically challenging variant; in a controlled ablation isolating JiraAnchor, categorical-value accuracy rises from 48.7% to 71.7%, with component-field accuracy jumping from 16.9% to 66.2%. Our analysis identifies inherent semantic ambiguities, such as issue-type disambiguation and text-field selection, as the dominant failure modes rather than value-resolution errors, pointing to concrete directions for future work. We publicly release the benchmark, all agent transcripts, and evaluation code to support reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。