让编码代理学会理解数据架构并自主执行分析任务。
CRAFT: Learn the Schema, Execute the Plan

- 分两阶段训练代理:先学结构化计划,再通过强化学习优化工具选择与代码质量。
- 相比传统方法,准确率提升9.6个百分点,输入token减少9倍,多轮对话更连贯。
- 适合需要稳定、低延迟企业级数据分析的智能代理系统开发者。
企业级编码代理将自然语言分析请求转化为针对专有API、数据模式和指标定义的可执行代码。现有方法在提示中注入完整模式信息,导致推理开销大、模式更新难、多轮分析可靠性差。本文提出CRAFT,一种基于后训练的两阶段方案:第一阶段采用无模式提示的监督微调,从已验证轨迹中学习领域结构化计划与可执行行为;第二阶段通过执行导向的强化学习,对工具选择、代码质量、计划-代码一致性及失败恢复能力进行联合优化。训练轨迹由三重过滤器(执行验证、数据完整性检查、LLM判断审计)筛选。评估覆盖广告分析场景,包括活动表现分析、指标下钻、实体级性能分析与多轮迭代优化。测试环境包含25个核心实体关联的模式与30条代理工作流。相比带模式填充的基线,CRAFT在综合代理评分上提升9.6个百分点,一致性提高4.1个百分点,多轮连贯性提升4.2个百分点,输入token量降低约9倍,模式发现循环减少最多5倍。还报告了部署权衡、奖励设计局限及企业级多轮工具使用强化学习所需的基础设施扩展。
原文摘要 · Abstract (English)
Enterprise coding agents translate natural-language analytical requests into executable code over proprietary APIs, schemas, and metric definitions. Yet the prevailing deployment pattern injecting exhaustive schema and tool documentation into each prompt increases inference overhead, complicates schema evolution, and undermines reliability in multi-turn analysis. We investigate whether stable schema knowledge and tool-use behavior can instead be acquired through post-training while preserving the consistency required for production-facing analytics. We present CRAFT, a two-stage post-training recipe for schema-grounded coding agents. First, schema-stripped PLAN supervised fine-tuning learns domain-structured plans and executable behaviors from validated trajectories without exhaustive prompt-time schema injection. Second, execution-shaped reinforcement learning aligns the policy for tool selection, code quality, plan-code consistency, and recovery from failed executions. Training trajectories are curated through a Tri-Gate filter combining execution validation, data-integrity checks, and LLM-judge reasoning audit. We evaluate CRAFT for planned rollout in advertising analytics, covering campaign performance analysis, metric drill-downs, entity-level performance analysis, and multi-turn analytical refinement. The enterprise evaluation environment incorporates beta APIs as the agent-facing tool surface and spans 25 schema-linked core entities and 30 agentic workflows. Relative to a schema-stuffed baseline, CRAFT improves composite Agent Score by +9.6 pp, consistency by +4.1 pp, and multi-turn coherence by +4.2 pp, while reducing input-token burden by approximately 9x and schema-discovery loops by up to 5x. We further report deployment tradeoffs, reward-shaping limitations, and training-infrastructure extensions required for multi-turn tool-use reinforcement learning in enterprise settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。