让AI写SQL更聪明:用新奖励机制解决反馈稀疏问题
SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation
- 设计两级奖励机制,分别处理多轮任务的信用分配和步骤级反馈
- 在BIRD数据集上比传统方法提升5%,超越SOTA模型表现
- 适合研究多轮自然语言转SQL或强化学习应用的开发者
代理强化学习在复杂任务中展现潜力,但文本转SQL仍局限于单轮模式。主要瓶颈是信用分配难题:传统方法仅根据最终结果给奖赏,忽略中间过程,导致评估模糊。为此,我们提出Agentic SQL框架,采用通用双层奖励机制实现轨迹级评估与密集步骤级信号。首先引入聚合轨迹奖励(ATR),通过非对称转移矩阵整合过程评分,激励持续改进;基于李雅普诺夫稳定性理论,证明ATR为能量耗散算子,保证无环策略与单调收敛。其次,列集匹配奖励(CSMR)提供即时步骤级奖励,每轮执行查询后将二元反馈(0/1)转化为[0,1]密度信号,反映部分正确性。在BIRD上,相比二值奖励的GRPO提升5%;使用相同模型时,优于SOTA Arctic-Text2SQL-R1-7B,在BIRD与Spider 2.0上均表现更优,推动文本转SQL向稳健多轮代理范式演进。
原文摘要 · Abstract (English)
Agentic Reinforcement Learning (RL) shows promise for complex tasks, but Text-to-SQL remains mostly restricted to single-turn paradigms. A primary bottleneck is the credit assignment problem. In traditional paradigms, rewards are determined solely by the final-turn feedback, which ignores the intermediate process and leads to ambiguous credit evaluation. To address this, we propose Agentic SQL, a framework featuring a universal two-tiered reward mechanism designed to provide effective trajectory-level evaluation and dense step-level signals. First, we introduce Aggregated Trajectory Reward (ATR) to resolve multi-turn credit assignment. Using an asymmetric transition matrix, ATR aggregates process-oriented scores to incentivize continuous improvement. Leveraging Lyapunov stability theory, we prove ATR acts as an energy dissipation operator, guaranteeing a cycle-free policy and monotonic convergence. Second, Column-Set Matching Reward (CSMR) provides immediate step-level rewards to mitigate sparsity. By executing queries at each turn, CSMR converts binary (0/1) feedback into dense [0, 1] signals based on partial correctness. Evaluations on BIRD show a 5% gain over binary-reward GRPO. Notably, our approach outperforms SOTA Arctic-Text2SQL-R1-7B on BIRD and Spider 2.0 using identical models, propelling Text-to-SQL toward a robust multi-turn agent paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。