Tahoe通过经验学习优化提示,显著提升Text-to-SQL准确率。
TAHOE: Text-to-SQL with Automated Hint Optimization from Experience
- 基于错误反馈构建结构化提示库,动态生成语法与语义提示
- 在Spider 2.0-Snow上将通过率从61.95%提升至79.42%
- 适用于生产级数据库系统,兼容弱模型且无需重训练
大语言模型(LLM)使文本转SQL成为可能,但投入生产仍面临挑战:需适配严格语法、处理复杂模式并适应用户偏好变化。监督微调成本高,代理式测试时扩展代价大。本文提出Tahoe,将提示优化视为动态数据管理问题。Tahoe在开发与部署阶段构建错误驱动的提示学习流程,将调试日志整合为结构化提示库。编译器反馈提炼为特定方言的语法提示,执行结果与用户反馈转化为模式和用户相关的语义提示。系统引入策略层,将冲突用户意图建模为共享自然语言触发下的竞争策略,结合时效信号与后学习归因统计,总结实际成功、伤害、无效与支持情况。推理时,Tahoe检索相关提示,通过逻辑规划引导大模型生成SQL。我们实现并评估了开发阶段工作流,部署阶段人工反馈更新留待后续。在Spider 2.0-Snow上,使用GPT-5.5,Tahoe将通过率从61.95%提升至79.42%,pass-at-4从72.57%升至87.61%,雪弗兰语法通过率达100%,平均编译器批评轮次由2.79降至0.12。同一提示库迁移至较弱模型(Doubao-2.0-lite)亦带来19.7个百分点的通过率提升。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have democratized database access through Text-to-SQL, but moving from prototypes to production remains difficult. Real deployments must handle strict SQL dialects, massive schemas, and evolving user preferences, while supervised fine-tuning is costly and rigid and agentic test-time scaling is expensive. We present Tahoe, a system that treats prompt optimization as a dynamic data management problem. Tahoe uses an error-driven hint learning pipeline across Development and Deployment to consolidate debugging traces into a structured Hint Bank. Compiler feedback is distilled into reusable Syntax Hints for dialect-specific rules, while execution and user feedback are converted into Semantic Hints for schema- and user-specific logic. Tahoe further introduces a Strategy Layer that models conflicting user intents as competing strategies under shared natural-language triggers, with recency signals and post-learning attribution statistics that summarize empirical success, harm, inertness, and support. At inference time, Tahoe retrieves relevant hints and guides the LLM through Logic Planning followed by SQL Synthesis. We implement and evaluate the development-phase workflow, leaving deployment-time human-feedback updates for future work. On Spider 2.0-Snow, Tahoe substantially improves Text-to-SQL without updating model parameters. On 113 supervised Spider 2.0-Snow-0212 examples using GPT-5.5, Tahoe raises pass rate from 61.95 percent to 79.42 percent and pass-at-4 from 72.57 percent to 87.61 percent, achieves 100 percent Snowflake syntax pass rate, and reduces average compiler-feedback critic rounds from 2.79 to 0.12 per sampled candidate. The same Hint Bank also transfers to weaker backbones, including a 19.7 percentage-point pass-rate gain on Doubao-2.0-lite.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。