arXiv:2606.18467stat.MLcs.LG2026-06被引 2

为使用工具和检索的AI代理设计了全程风险控制方法,防止错误累积。

ToolChain-CRC: Conformal Risk Control for Agentic AI Under Retrieval and Tool-Use Drift

论文配图:ToolChain-CRC: Conformal Risk Control for Agentic AI Under Retrieval and Tool-Use Drift
图 1 · 摘自论文原文
  • 将代理运行视为完整动作轨迹,逐步评估风险并合成总风险
  • 在多种测试中,传统只看最终答案的方法会漏掉错误,而该方法能保持风险低于目标
  • 支持随时报警干预,适合高可靠性场景的AI系统

现代AI代理通过检索文档、调用工具、检查中间信息生成最终回答或动作,这一过程存在仅看最终输出无法察觉的风险。即使检索薄弱、工具输出错误或前期步骤无依据,最终结果也可能看似合理。本文提出ToolChain-CRC,一种针对检索增强与工具使用代理在漂移下的共形风险控制方法。该方法将每次代理运行视为完整动作、观测与输出的轨迹,构建步骤级风险评分,整合为轨迹风险评分,校准接受或干预规则,并加入可随时触发的预警机制,在最终输出前拦截高风险运行。理论证明在交换性校准运行下实现轨迹级风险控制,提出具有可审计常数的漂移感知扩展,并通过超鞅构造证明任意时间升级规则。实验涵盖合成工具链漂移、RAG/工具使用压力测试、基于SQuAD的检索任务、无API代理问答案例研究、消融实验、目标风险敏感性检验、20次种子鲁棒性测试、漂移裕度审计及实时RAG/工具使用代理基准测试。结果表明,仅基于最终答案的校准会遗漏检索与工具失败,而轨迹级校准始终将被接受轨迹的风险控制在目标以下。

原文摘要 · Abstract (English)

Modern AI agents retrieve documents, call tools, check intermediate information, and then produce a final answer or action. This creates a risk-control problem that is not visible from the final answer alone. A final response may look acceptable even when the retrieval was weak, a tool output was wrong, or an earlier step was unsupported. We propose ToolChain-CRC, a conformal risk-control method for retrieval-augmented and tool-using agents under drift. The method treats each agent run as a full trajectory of actions, observations, and final output. It builds step-level risk scores, combines them into a trajectory risk score, calibrates an accept-or-intervene rule, and adds an anytime alarm that can stop risky runs before the final answer. We prove trajectory-level risk control under exchangeable calibration runs, give a drift-aware extension with auditable constants, and prove an anytime escalation rule through a supermartingale construction. Experiments cover synthetic tool-chain drift, RAG/tool-use stress tests, public SQuAD-derived retrieval tasks, an API-free agentic QA case study, ablations, target-risk sensitivity checks, 20-seed robustness checks, a drift-margin audit, and a live RAG/tool-use agent benchmark. Across these settings, final-answer-only calibration can miss retrieval and tool failures, while trajectory-level calibration keeps accepted-trajectory risk below the target.

AI安全风险控制工具使用检索增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。