arXiv:2605.06890cs.AIcs.MA2026-05被引 1

用内部信号提前预判AI工具调用风险,解决长流程中错误累积问题。

Beyond the Black Box: Interpretability of Agentic AI Tool Use

论文配图:Beyond the Black Box: Interpretability of Agentic AI Tool Use
图 1 · 摘自论文原文
  • 通过稀疏自编码器和线性探针分析模型内部激活,识别工具调用决策依据。
  • 在NVIDIA Nemotron数据集上实现对工具调用必要性和风险的提前预测。
  • 适用于需要高可靠性的企业级AI代理系统,尤其适合长流程任务监控。

AI代理在高风险企业工作流中前景广阔,但其工具调用决策难以诊断与控制,导致可能跳过必要调用、冗余调用或执行后才暴露后果。现有可观测性方法均为外部:提示词揭示相关性,评估打分输出,日志仅在行动后生成。在长周期场景中,早期工具错误会改变后续执行轨迹,增加令牌消耗,并引发下游安全与风险。本文提出基于稀疏自编码器(SAEs)与线性探针的机制可解释性工具包,将模型激活分解为稀疏内部特征,读取每个动作前的状态,推断是否需要工具及下一步操作的风险。该框架识别与工具决策最相关的层与特征,并通过特征消融测试其功能重要性。在NVIDIA Nemotron函数调用数据集的多步代理执行轨迹上训练探针,并应用于GPT-OSS 20B与Gemma 3 27B模型。目标并非取代外部评估,而是补充缺失的一环:行动前模型内部信号的可见性。这有助于揭示代理失败的深层原因,尤其在长周期运行中早期错误影响后续行为的情况。更广泛而言,论文展示了机制可解释性如何支持代理系统中工具调用与风险的内部可观测性。

原文摘要 · Abstract (English)

AI agents are promising for high-stakes enterprise workflows, but dependable deployment remains limited because these tool-use decisions are difficult to diagnose and control. Agents may skip required tool calls, invoke tools unnecessarily, or take actions whose consequences become visible only after execution. Existing observability methods are external: prompts reveal correlations, evaluations score outputs, and logs arrive only after the model has already acted. In long-horizon settings, these failures are costly because an early tool mistake can alter the rest of the execution trajectory, increase token consumption, and create downstream safety and security risk. We introduce a mechanistic-interpretability toolkit built on Sparse Autoencoders (SAEs), which decompose activations into sparse internal features, and linear probes, lightweight classifiers that read signals from those features. The framework reads model states before each action and infers whether a tool is needed and how risky the next tool action is. It identifies the model layers and features most associated with tool decisions and tests their functional importance through feature ablation. We train the probes on multi-step agent execution traces from the NVIDIA Nemotron function-calling dataset and apply the same workflow to GPT-OSS 20B and Gemma 3 27B models. The goal is not to replace external evaluation, but to add a missing layer: visibility into what the model signaled internally before action. This helps surface deeper causes of agent failure, especially in long-horizon runs where an early mistake can impact subsequent agent behavior. More broadly, the paper shows how mechanistic interpretability can support internal observability for monitoring tool calls and risk in agent systems.

AI代理可解释性工具调用风险预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。