揭示大模型在代理系统中的隐藏风险，提出可量化评估的新方法
Mind the Gap: Evaluating Model- and Agentic-Level Vulnerabilities in LLMs with Action Graphs
- 用动作图分解代理行为，精准追踪风险点
- 发现代理特有漏洞，工具调用使攻击成功率升24%-60%
- 自动优化提示词，有效降低代理越狱成功率
随着大语言模型广泛应用于代理系统，现有评估方法在观测、评估和缓解部署风险方面存在显著缺口。本文提出以可观测性为核心的全流程方法：引入 extbf{AgentSeer}工具，将代理执行过程分解为细粒度的行动-组件图；基于此，通过跨模型验证（GPT-OSS-20B与Gemini-2.0-flash）在HarmBench上量化了模型级与代理级越狱风险之间的差距，涵盖单轮与迭代优化攻击。结果表明，模型级评估显示：GPT-OSS-20B（ASR 39.47%）与Gemini-2.0-flash（ASR 50.00%）均易受社会工程攻击。而代理级评估暴露了传统方法无法捕捉的“代理专属”风险，工具调用导致两模型攻击成功率提升24%-60%。跨模型分析揭示通用模式：代理转移操作为最高风险工具，其风险源于语义模式而非语法漏洞。直接攻击从模型级迁移至代理级后，成功提示性能下降（GPT-OSS-20B：57%人类注入ASR；Gemini-2.0-flash：28%），而上下文感知的迭代攻击却能攻破模型级失败的目标，证实系统性风险鸿沟。基于动作信号的提示词优化显著降低代理级越狱成功率（GPT-OSS-20B：直接攻击降至45.3%）。
原文摘要 · Abstract (English)
As large language models increasingly deployed into agentic systems, existing methods face critical gaps in observing, assessing, and mitigating deployment-specific risks. We present a comprehensive, observability-driven workflow: we introduce \textbf{AgentSeer}, observability tool which decomposes agentic executions into granular \emph{action-component} graphs; we use this decomposition to rigorously quantify the gap between model-level and agent-level jailbreaking risk via cross-model validation on GPT-OSS-20B and Gemini-2.0-flash with HarmBench under single-turn and iterative-refinement attacks; we leverage action-graph risk signals to automate iterative prompt hardening against direct and iterative jailbreak attacks. Stark differences is revealed between model-level and agentic-level vulnerability profiles. Model-level evaluation reveals baseline differences: GPT-OSS-20B (39.47\% ASR) versus Gemini-2.0-flash (50.00\% ASR), with both models showing susceptibility to social engineering. However, agentic-level assessment exposes agent-specific risks invisible to traditional evaluation. We discover "agentic-only" vulnerabilities that emerge exclusively in agentic contexts, with tool-calling showing 24-60\% higher ASR across both models. Cross-model analysis reveals universal agentic patterns, where agent transfer operations as highest-risk tools, with semantic pattern revealed rather than syntactic vulnerability mechanisms. Direct attack transfer from model-level to agentic contexts shows degraded performance of successful prompts (GPT-OSS-20B: 57\% human injection ASR; Gemini-2.0-flash: 28\%), while context-aware iterative attacks successfully compromise objectives that failed at model-level, confirming systematic vulnerabilities gaps. Action-based prompt improvement substantially reduces action-averaged agentic jailbreak success on GPT-OSS-20B (direct: 45.3\%
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。