让文本转SQL系统学会识别无法回答的问题,提升生物医学查询的可靠性。
Query Carefully: Detecting the Unanswerables in Text-to-SQL Tasks
- 用提示词和规则结合大模型,判断自然语言查询是否可答。
- 在80个无答案问题上检测准确率达80%,结构类问题几乎全对。
- 适合医疗、科研等对准确性要求高的场景使用。
文本转SQL系统使非专业用户能通过自然语言操作数据库,但其常对模糊、超范围或无法回答的查询生成可执行的SQL,带来误判风险,尤其在生物医学领域危害严重。为此,我们提出Query Carefully,将大模型生成与无答案检测相结合。基于ScienceBenchmark的OncoMX组件,构建了包含80个无答案问题的OncoMX-NAQ数据集,涵盖8类(非SQL、超出模式/领域、多种模糊类型)。方法采用llama3.3:70b,结合上下文感知提示、显式无答案规则(NAR)及来自可答与不可答问题的少量示例。评估包括SQL精确匹配、结果准确率和无答案检测准确率。在OncoMX开发集上,仅用可答示例的少样本提示可提升结果准确率,加入不可答示例不降低性能。在OncoMX-NAQ上,均衡提示下无答案检测准确率达0.8,结构类问题(非SQL、缺字段、跨领域)接近完美,但缺失值查询(0.5)和列歧义(0.3)仍存挑战。轻量级界面展示中间生成的SQL、执行结果与放弃回答,支持透明可靠的生物医学文本转SQL应用。
原文摘要 · Abstract (English)
Text-to-SQL systems allow non-SQL experts to interact with relational databases using natural language. However, their tendency to generate executable SQL for ambiguous, out-of-scope, or unanswerable queries introduces a hidden risk, as outputs may be misinterpreted as correct. This risk is especially serious in biomedical contexts, where precision is critical. We therefore present Query Carefully, a pipeline that integrates LLM-based SQL generation with explicit detection and handling of unanswerable inputs. Building on the OncoMX component of ScienceBenchmark, we construct OncoMX-NAQ (No-Answer Questions), a set of 80 no-answer questions spanning 8 categories (non-SQL, out-of-schema/domain, and multiple ambiguity types). Our approach employs llama3.3:70b with schema-aware prompts, explicit No-Answer Rules (NAR), and few-shot examples drawn from both answerable and unanswerable questions. We evaluate SQL exact match, result accuracy, and unanswerable-detection accuracy. On the OncoMX dev split, few-shot prompting with answerable examples increases result accuracy, and adding unanswerable examples does not degrade performance. On OncoMX-NAQ, balanced prompting achieves the highest unanswerable-detection accuracy (0.8), with near-perfect results for structurally defined categories (non-SQL, missing columns, out-of-domain) but persistent challenges for missing-value queries (0.5) and column ambiguity (0.3). A lightweight user interface surfaces interim SQL, execution results, and abstentions, supporting transparent and reliable text-to-SQL in biomedical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。