不执行代码就提前检测工具调用错误，提升AI代理可靠性
RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement
- 用任务定制的检查清单预审代码，提前发现工具间接口错误
- 在M3ToolEval上准确率达0.86，比之前方法高且无需实际运行
- 适合多工具协作场景，尤其擅长识别输出形状、参数传递等缺陷
迭代自修正是一种流行的推理阶段可靠性技术，但在代码模式的工具使用中，其效果高度依赖反馈信号结构：非结构化批评作用不稳定，即使使用真实执行反馈也仅小幅提升（0.75 vs. 0.65基线）。主要失败原因是跨工具合约违规（输出形状错误、工具路由错误、参数来源断裂），这些错误可完整运行却不触发异常，导致运行时反馈无效。我们提出RubricRefine，一种无需训练的预执行语义合约验证方法：生成任务与工具注册表特定的检查标准，对候选代码进行显式合约评分，并在执行前迭代修复问题。RubricRefine在七种模型上平均达到0.86的准确率（M3ToolEval），且零次执行尝试，相比先前推理时基线最高降低2.6倍延迟。在以单步调用为主的API-Bank上表现持平，符合该方法依赖跨工具合约结构的特性。通过检查类别消融和校准分析，进一步揭示了该方法适用边界与生效原因。
原文摘要 · Abstract (English)
Iterative self-refinement is a popular inference-time reliability technique, but its effectiveness in code-mode tool use depends heavily on the structure of the feedback signal: unstructured critique helps inconsistently across models, and even revision with real execution feedback improves only modestly ($0.75$ vs. $0.65$ baseline). The dominant failures are inter-tool contract violations (wrong output shape, incorrect tool routing, broken argument provenance) that run to completion without raising errors, making runtime feedback insufficient. We introduce RubricRefine, a training-free method for pre-execution semantic contract verification that generates task- and registry-specific rubrics, scores candidate code against explicit contract checks, and iteratively repairs failures before any execution occurs. RubricRefine reaches $0.86$, averaged across seven models, on M3ToolEval with zero execution attempts, improving over prior inference-time baselines with up to $2.6\times$ lower latency. Performance remains flat on the predominantly single-step API-Bank, consistent with the method's reliance on inter-tool contract structure. A rubric-category ablation and calibration analysis further characterize when and why the method works.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。