用约束强化学习提升文本转SQL模型，防止奖励欺骗
ConstrainedSQL: Training LLMs for Text2SQL via Constrained Reinforcement Learning
- 引入可解释的奖励与约束信号，动态平衡训练目标
- 在Text2SQL基准数据集上优于现有强化学习方法
- 适合需要高精度和可靠性的数据库查询生成任务
强化学习（RL）在提升文本转SQL大模型推理能力方面展现出巨大潜力，尤其在使用GRPO和DAPO等先进算法时。然而，这些方法的性能对奖励函数设计极为敏感，不当的奖励可能导致奖励欺骗，即模型通过利用奖励结构漏洞获得高分，而非真正解决任务。本文提出一种用于文本转SQL的约束强化学习框架，融合自然且可解释的奖励与约束信号，并在训练过程中动态调节它们之间的权衡。我们建立了该约束强化学习框架的理论保证，并在多个知名文本转SQL数据集上的实验验证了该方法相较于当前最先进的强化学习训练大模型的显著提升。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has demonstrated significant promise in enhancing the reasoning capabilities of Text2SQL LLMs, especially with advanced algorithms such as GRPO and DAPO. However, the performance of these methods is highly sensitive to the design of reward functions. Inappropriate rewards can lead to reward hacking, where models exploit loopholes in the reward structure to achieve high scores without genuinely solving the task. This work considers a constrained RL framework for Text2SQL that incorporates natural and interpretable reward and constraint signals, while dynamically balancing trade-offs among them during the training. We establish the theoretical guarantees of our constrained RL framework and our numerical experiments on the well-known Text2SQL datasets substantiate the improvement of our approach over the state-of-the-art RL-trained LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。