arXiv:2608.23911cs.AIcs.LG2026-08

通过优化失败轨迹提升模型蒸馏效果,显著改善工具调用能力。

PROOF-Gen: From Optimized Data to Better Distillation

  • 针对失败任务进行逐场景提示优化,从错误中恢复正确轨迹。
  • 在τ2-bench上将失败场景恢复率提升至93%,模型通过率提高至52.9%。
  • 适用于需要高可靠性工具调用的部署场景,尤其适合资源受限设备。

基于教师生成轨迹的监督微调是将工具调用能力蒸馏到可部署模型的标准第一阶段。后训练流水线每日或每周重复此阶段,需持续支付前沿教师成本,但现有机制为生成-过滤(保留通过轨迹,丢弃其余),且每次循环均遗留相同难题,因失败不提供信号。在τ2-bench上,57%的教师尝试失败,其中三分之二为近似成功(多数工具调用正确,仅因一个关键错误失败)。我们提出PROOF-Gen(Per-scenario Reflective Optimization to Overcome Failed Generation),通过逐场景提示优化从失败中恢复黄金轨迹。对每个失败任务,反射器分析执行日志与评估反馈,生成修正指导,引导教师生成通过轨迹。指导在训练前被移除,学生仅学习干净示范,无任务特定辅助。在τ2-bench上,逐场景优化恢复率达93%。在合并数据上微调,Qwen3-4B-Instruct-2507的通过率从0.132提升至0.529;Gemma 4 E4B-it在BFCL v4多轮任务上提升7.2pp。在部署管道中,轨迹质量提升6.3pp目标完成率,并迁移至本地设备模型(目标完成率+1.5pp;响应质量指标提升+1.7至+5.0pp),所有地区均实现正向迁移(非英语平均+1.48pp)。

原文摘要 · Abstract (English)

Supervised fine-tuning on teacher-generated trajectories is the standard first stage for distilling tool-calling capabilities into deployable models. Post-training pipelines that drive shipped tool-calling agents re-run this stage on a daily or weekly cadence, paying the frontier-teacher cost each cycle, yet the mechanism is generate-and-filter (keep the teacher's passing trajectories, discard the rest) and each cycle leaves behind the same hard scenarios because failures supply no signal. On τ2-bench, 57% of teacher trials fail, two-thirds of them near-misses (most tool calls correct, undone by one decisive error). We introduce PROOF-Gen (Per-scenario Reflective Optimization to Overcome FailedGeneration), which recovers golden trajectories from these failures via per-scenario prompt optimization. For each failed task, a reflector analyzes the execution trace and evaluation feedback, then writes corrective guidance that steers the teacher to a passing trajectory. The guidance is stripped before training, so the student learns from clean demonstrations with no task-specific scaffold. On τ2-bench, per-scenario optimization recovers 93% of failed scenarios. Fine-tuned on the combined data, Qwen3-4B-Instruct-2507 improves from Pass^1=0.132 to 0.529 and Gemma 4 E4B-it gains +7.2pp on BFCL v4 multi-turn. In a deployed pipeline, the method lifts trajectory quality by +6.3pp goal completion and transfers to a deployed on-device model (+1.5pp goal completion; +1.7 to +5.0pp across response-quality metrics), with positive transfer in every locale (non-English average +1.48pp).

模型蒸馏工具调用轨迹优化部署增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。