用可验证奖励训练小模型,让语言模型在企业系统中准确调用API。
Beyond Next-Token Prediction: An RLVR Proof of Concept for Tool-Use Agents on Atlassian Workflows
- 直接在真实API环境里用可验证奖励进行强化学习。
- 小模型在五种场景中平均得分从0.35提升至1.00,页面创建提升最明显。
- 适合想优化特定企业系统工具调用的开发者和研究者。
大型语言模型通常训练为预测下一个词元,而非在特定API环境中执行操作。在专业企业SaaS工作流中——成功意味着以正确顺序调用正确的嵌套参数端点——这种目标错位表现为无声失败:缺失必填字段、虚构工具或仅读取一次就提前终止。我们探讨是否可通过在目标环境中直接应用可验证奖励的强化学习(RLVR)来弥合这一差距。作为概念验证,我们构建了五个合成环境,模拟Jira REST v3与Confluence v2 API的模式精度;奖励完全基于工具调用轨迹计算,无需真实API、无需学习型裁判,也无人工标注。在与GRPO训练相同的检测器上评估提示型Qwen3-1.7B与Qwen3.5-4B,发现在四个非退化奖励场景中,强化学习训练后的策略将平均奖励从基线0.35–0.92提升至0.95–1.00,其中Confluence页面创建提升最大(0.35 → 1.00)。这标志着向针对特定企业API实现结果优化的小模型迈出初步一步,并指出两个需权衡的局限:可验证奖励的手工设计难以扩展到更多端点,且其中一个场景(票务流转)的奖励函数饱和,4B提示模型已达到上限。
原文摘要 · Abstract (English)
Large language models are trained to predict the next token, not to act inside a specific API. In niche enterprise SaaS workflows -- where success means hitting the right endpoint with the right nested arguments in the right order -- this objective mismatch shows up as silent failures: dropped required fields, hallucinated tools, or early stops after a single read. We ask whether Reinforcement Learning with Verifiable Rewards (RLVR), applied directly in the target environment, closes the gap. As a proof of concept we build a suite of five synthetic environments emulating the Jira REST v3 and Confluence v2 APIs at schema fidelity; rewards are computed entirely from the tool-call trace, with no live API, no learned judge, and no human label in the loop. Scoring prompted Qwen3-1.7B and Qwen3.5-4B on the same checkers that drive GRPO training, we find that on the four scenarios whose rewards are non-degenerate the RL-trained policy lifts average reward from a 4B-baseline range of 0.35--0.92 to 0.95--1.00, with the largest single gain on Confluence page creation ($0.35 \rightarrow 1.00$). We position this as a preliminary step toward outcome-optimised small models for niche enterprise APIs, and foreground two limitations a workshop reader should weigh: hand-crafting verifiable rewards does not scale beyond the handful of endpoints reported here, and one of our five scenarios (ticket-transition) has a saturating reward shape that the prompted 4B already maxes out.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。