用数据库执行反馈训练模型，提升自然语言转SQL的准确率
Reinforcing Code Generation: Improving Text-to-SQL with Execution-Based Learning
- 通过数据库执行结果作为奖励信号，用强化学习优化代码生成
- 准确率从31.49%提升至49.83%，错误率降至14.71%
- 仅需问题-答案对即可接近70B大模型性能，适合低资源场景
本文研究大语言模型在代码生成任务中的应用，聚焦于从自然语言问题生成SQL查询。我们提出：能否不依赖文本-代码成对监督微调，而是让模型与数据库引擎交互来学习？将该问题建模为强化学习任务，模型从执行环境中获得标量奖励——执行失败惩罚，正确返回则获正奖励。采用组相对策略优化（GRPO）框架进行训练。在表格推理基准上测试发现，仅使用问题-答案对这一弱监督信号，模型生成的SQL准确率由31.49%提升至49.83%，错误率从25.43%降至14.71%。该性能几乎媲美更大的SQLCoder-70B模型。本工作展示了利用执行反馈提升大模型符号推理能力的巨大潜力。
原文摘要 · Abstract (English)
In this work, we study the problem of code generation with a large language model (LLM), with a focus on generating SQL queries from natural language questions. We ask: Instead of using supervised fine tuning with text-code pairs, can we tune a model by having it interact with a database engine? We frame this problem as a reinforcement learning problem where the model receives execution-based feedback from the environment in the form of scalar rewards. These rewards penalize execution failures and assign positive values when a query returns a correct answer. We use the rewards within the Group Relative Policy Optimization (GRPO) framework. We use a tabular reasoning benchmark to test and evaluate our findings. We find that with only weak supervision in the form of question-answer pairs, RL-tuning improves the accuracy of model generated SQL code from 31.49 to 49.83 while reducing error percentage from 25.43% to 14.71%. This improvement allowed the model nearly match the performance performance to the larger SQLCoder-70B model. Our work demonstrates the potential of using execution-based feedback to improve symbolic reasoning capabilities of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。