让大模型生成SQL时每一步都得分,避免无效操作。
Every Step Counts: Step-Level Credit Assignment for Tool-Integrated Text-to-SQL

- 为每一步推理设计独立奖励,精准反馈中间过程
- 在BIRD数据集上提升3.25%执行准确率,减少冗余调用
- 适合想优化复杂SQL生成效率的研究者和开发者
工具增强的文本转SQL已成为一种有前景的范式,将SQL生成视为与工具执行交错的序列决策过程。然而,现有强化学习方法主要依赖粗粒度的结果监督,导致根本性的信用分配问题:只要最终答案正确,无论中间步骤是否冗余、低效或错误,模型都获得相同奖励。这促使模型探索次优推理空间,限制了效率与泛化能力。为此,我们提出FineStep框架,实现工具增强型Text-to-SQL的步级信用分配。首先,引入独立过程奖励以缓解结果监督下的信号稀疏性;其次,提出步级信用分配机制,精确量化每一步推理的价值;最后,设计基于步级优势的策略优化方法,实现高效更新。在BIRD基准上的大量实验表明,FineStep达到当前最优性能,在4B规模下相比GRPO平均提升3.25%的执行准确率(EX),并显著减少冗余工具交互。
原文摘要 · Abstract (English)
Tool-integrated Text-to-SQL parsing has emerged as a promising paradigm, framing SQL generation as a sequential decision-making process interleaved with tool execution. However, existing reinforcement learning approaches mainly rely on coarse-grained outcome supervision, resulting in a fundamental credit assignment problem: models receive the same reward for any trajectory that yields the correct answer, even when intermediate steps are redundant, inefficient, or erroneous. Consequently, models are encouraged to explore suboptimal reasoning spaces, limiting both efficiency and generalization. To address this problem, we propose FineStep, a novel framework for step-level credit assignment in tool-augmented Text-to-SQL. First, we introduce a reward design with independent process rewards to alleviate the signal sparsity of outcome supervision. Next, we present a step-level credit assignment mechanism to precisely quantify the value of each reasoning step. Finally, we develop a policy optimization method based on step-level advantages for efficient updates. Extensive experiments on BIRD benchmarks show that FineStep achieves state-of-the-art performance and reduces redundant tool interactions, with a 3.25% average EX gain over GRPO at the 4B scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。