arXiv:2605.17453cs.CRcs.CL2026-05被引 1

研究大模型工具调用中的信任陷阱,提出防御新方法

Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback

论文配图:Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback
图 1 · 摘自论文原文
  • 将多步工具交互抽象为信任动态变量,追踪信任形成过程
  • 在1970个隐蔽触发场景中,新方法在域内达84.2分,跨域仍保持56.9分
  • 突破传统提示或工具描述的防御局限,关注信任累积轨迹

工具使用的大语言模型代理日益依赖外部工具做出重要决策,但现有安全评估与防御大多假设选定工具的反馈始终可信。本文研究一种新型失效模式——认知污染:恶意工具在探索阶段表现合理,通过看似无害的反馈积累信任,仅在隐藏状态条件与最终执行动作对齐时才显露出危害。为此,我们构建了TRUST-Bench基准,包含1,970个任务条件化的隐蔽触发工具劫持场景,并配备匹配的安全对照组;引入非对称惩罚指标GuardedJoint以更真实反映部署风险;提出VISTA-Guard框架,实现对最终执行动作的风险评分。核心思想是将多步工具交互抽象为结构化环境变量,编码信任形成动态,并基于此轨迹条件表示评估最终动作风险。实验表明,提示中心启发式、标量特征和零样本判断器在此场景下失效,而轨迹感知的最终动作评分展现出强域内区分能力,并在平衡的域外迁移中依然有效。在GuardedJoint指标下,VISTA-Guard在域内得分为84.2,在平衡域外为56.9,而仅优化安全或效用单一维度的方法则降至零。这些发现支持对黑箱工具生态中代理安全的更广视角:关键防御目标并非局部提示文本或工具描述,而是信任在整个交互轨迹中的形成方式及其通过最终动作的固化。

原文摘要 · Abstract (English)

Tool-using LLM agents increasingly rely on external tools to make consequential decisions, yet most existing agent-security benchmarks and defenses implicitly assume that tool feedback is trustworthy once a tool has been selected. We study a different failure mode, cognitive poisoning, in which a malicious tool behaves plausibly during exploration, accumulates trust through benign-looking feedback, and becomes harmful only when hidden state conditions align with the final executable action. To study this setting, we construct TRUST-Bench, a task-conditioned benchmark of 1,970 hidden-trigger tool-compromise episodes with matched safe controls, introduce an asymmetric penalty metric, GuardedJoint, to better reflect real deployment risk, and present VISTA-Guard, a backbone-agnostic framework for final-action risk scoring. The core idea is to abstract multi-step tool interaction into structured environment variables that encode trust-formation dynamics and then score the risk of the final executable action from this trajectory-conditioned representation. Experiments show that prompt-centric heuristics, scalarized features, and zero-shot judges fail in this regime, whereas trajectory-aware final-action scoring yields strong in-domain discrimination and remains effective under balanced out-of-distribution transfer. Under GuardedJoint, VISTA-Guard reaches $84.2$ in-domain and $56.9$ on balanced out-of-distribution evaluation, while methods that optimize only one side of the safety--utility tradeoff collapse to zero. These findings support a broader view of agent security in black-box tool ecosystems: the decisive defense target is not local prompt text or tool descriptors alone, but the way trust is formed across the interaction trajectory and committed through the final action.

大模型安全工具调用信任机制风险评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。