arXiv:2505.04671cs.CLcs.LG2025-05被引 24

通过分步执行感知与过程奖励,提升复杂SQL生成准确率

Reward-SQL: Boosting Text-to-SQL via Stepwise Execution-Aware Reasoning and Process-Supervised Rewards

  • 分步构建SQL,用中间视图验证和结构化CTE逐步推理
  • 引入过程奖励模型,实现每步执行反馈的细粒度监督
  • 适合需要高精度、可解释性SQL生成的研究与应用

基于强化学习的大语言模型在文本转SQL任务中表现提升,但仍因两大局限而难以处理复杂查询:缺乏基于数据库反馈的分步执行感知推理,以及缺少指导推理优化的过程级奖励。为此,我们提出CoCTE框架,通过分治与执行感知推理,逐步构造带有中间视图验证和结构化通用表表达式(CTE)的SQL,提升准确率与可解释性。在此基础上,我们开发了三阶段统一方法Reward-SQL:(1) 模型初始化,赋予大模型结构化CoCTE推理能力;(2) 过程奖励设计,提供细粒度、执行感知的监督信号;(3) 过程监督强化学习与推理,将过程奖励融入训练并指导推理解码。本文提出过程奖励模型(PRM),结合执行感知轨迹评分与基于熵的步骤加权,实现全推理步骤的密集且可解释的监督。该模型同时应用于训练与推理阶段,稳定优化并增强轨迹探索。实验表明,Reward-SQL在模型规模相当的情况下显著优于基线,并具备强跨领域泛化能力。

原文摘要 · Abstract (English)

Recent advances in large language models (LLMs) trained with reinforcement learning (RL) have improved Text-to-SQL performance. However, RL-based approaches still struggle with complex queries due to two key limitations: insufficient stepwise execution-aware reasoning grounded in database feedback, and the lack of process-level rewards for guiding reasoning optimization. To address these issues, we propose CoCTE, a divide-and-conquer and execution-aware reasoning framework that progressively composes SQL queries through intermediate view validation and structured Common Table Expressions (CTEs), improving both accuracy and interpretability. To realize a CoCTE reasoning process, we develop Reward-SQL, a unified approach with three stages: (1) model initialization, which equips LLMs with structured CoCTE reasoning capabilities; (2) process reward design, which delivers fine-grained, execution-aware supervision; and (3) process-supervised RL and inference, which integrates process rewards into training and guides the inference stage by process rewards. This paper addresses the core challenges in Reward-SQL and makes the following contributions. We introduce a process reward model (PRM) that combines execution-aware trajectory scoring with entropy-based step weighting, providing dense and interpretable supervision across reasoning steps. We integrate PRM into both RL training and inference stages, stabilizing optimization and improving trajectory exploration with process-level signals. Experiments show that Reward-SQL significantly outperforms baselines with comparable model sizes, and exhibits strong cross-domain generalization.

文本转SQL强化学习可解释性推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。