在本地部署的开源大模型上，实测Text-to-SQL效果与优化策略有效性。
How Far Do On-Prem Open LLMs Get on Text-to-SQL? A Cross-Family Size x Technique Frontier on BIRD
- 统一协议对比三类开源模型，分步验证提示优化技巧
- 新一代模型性能远超旧版，自纠正几乎免费提升准确率
- 链接模式无效，自一致性代价高且收益微弱，适合追求性价比者
随着企业无法将数据外传至云端API，一个关键问题浮现：当模型必须在本地运行且使用开放权重时，Text-to-SQL的效果如何？哪些常见提升方法值得投入计算资源？本文在BIRD开发集（n=1534，执行准确率）上进行一次完整可复现的基准测试，评估了三类开源模型家族（Qwen2.5-Coder: 7B/14B/32B；CodeLlama-Instruct: 7B/13B/34B；Llama-3.x: 8B, 70B）在两代技术中的表现。通过统一协议，逐项消融一种通用优化方案（模式链接、自我纠错、自我一致性），所有差异均采用配对McNemar检验。主要发现：(i) 生成代际比模型规模更重要，且该优化方案具有家族鲁棒性：同规模下Qwen2.5-Coder（7B）优于CodeLlama（39.1 vs 20.9），而现代非Qwen模型（Llama-3.3-70B，49.2）也具备竞争力，说明CodeLlama的短板源于其2023年技术代际，而非“非Qwen即弱”。(ii) 自我纠错是稳定且近乎零成本的增益，在三类模型中均有显著效果。(iii) 模式链接无效，即使强链接器（金表召回率达96.5%）也无法改善，统计上等同于无链接，彻底排除“弱词法基线”争议。(iv) 自我一致性性价比极低（+0.13个百分点，耗时约5倍，不显著）。研究还报告各阶段真实成本（$/1000查询），并公开全部代码、预测结果和摘要；数据与代码已归档于 https://doi.org/10.5281/zenodo.20952794。
原文摘要 · Abstract (English)
Organizations that cannot send data to a cloud API increasingly ask: how good is Text-to-SQL if the model must run on-premises on open weights, and which popular accuracy "recipes" are worth their compute? We answer with an honest, fully reproducible benchmark on the BIRD development split (n=1534, Execution Accuracy), evaluating three open model families across two generations -- Qwen2.5-Coder (7B/14B/32B), CodeLlama-Instruct (7B/13B/34B), and Llama-3.x (8B, 70B) -- under one matched protocol, ablating a model-agnostic recipe (schema linking, self-correction, self-consistency) component by component, with every difference tested by the paired McNemar test. Four findings stand out. (i) Generation matters more than raw size, and the recipe is family-robust: Qwen2.5-Coder dominates the older CodeLlama at matched size (39.1 vs 20.9 at 7B), but a modern non-Qwen model (Llama-3.3-70B, 49.2 on a matched serving) is competitive, so CodeLlama's weakness reflects its 2023 generation, not "non-Qwen = weak". (ii) Self-correction is a robust, near-free win, significant on all three families where there is room to improve. (iii) Schema linking does not help, and a stronger linker does not rescue it: a retrieval/embedding linker with 96.5% gold-table recall is statistically indistinguishable from no linking, ruling out the "weak lexical strawman" objection across three families. (iv) Self-consistency is poor value (+0.13 pp for ~5x tokens, not significant). We report real per-stage cost ($/1k queries) and release all code, predictions, and summaries; archived code and data: https://doi.org/10.5281/zenodo.20952794
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。