arXiv:2604.15579cs.SEcs.AI2026-04被引 1

用符号规则保障AI代理安全,比训练模型更可靠

Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agents

  • 提出基于软件工程的符号规则,可形式化验证安全要求
  • 74%的安全需求可通过低成本检查实现,95%仅需简单校验
  • 适合高风险场景下部署的商业级AI代理,如医疗、金融系统

将调用工具的AI代理集成到领域特定商业软件中,意外的工具调用可能引发严重安全与安全事件。现有研究多依赖训练方法和神经防护机制,虽降低风险但无法保证杜绝。本文通过三部分研究:首先系统分析80个安全评测基准,发现85%未定义可验证需求(61%无具体要求,24%仅有高层目标);其次在τ²-Bench、CAR-bench和MedAgentBench上分析符号规则的适用性,发现74%的需求可被符号化强制,且95%仅需简单低开销检查;最后实证评估表明,符号规则在不损失实用性的前提下显著提升安全性与可靠性。研究呼吁重视符号化防护,为风险敏感型商业软件中的领域专用AI代理提供可信赖路径。代码与资源已开源。

原文摘要 · Abstract (English)

There is increasing interest in integrating AI agents that invoke tools into domain-specific commercial software, where unintended tool calls can cause serious security and safety incidents. This has drawn growing research attention, and many agent security and safety benchmarks have emerged. They implicitly shape how the community approaches security and safety. Yet existing work exhibits a blind spot: it emphasizes training-based methods and neural guardrails, which reduce the likelihood of insecure or unsafe actions but cannot guarantee their prevention. It generally overlooks opportunities for deductive, symbolic guardrails grounded in standard software engineering practices, which can provide guarantees for some security and safety requirements. Our study has three parts: (1) a systematic review of 80 agent security and safety benchmarks finding that that 85\% of benchmarks do not state verifiable requirements (61\% provide none, and 24\% give only high-level goals); (2) an applicability analysis of which security and safety requirements symbolic guardrails can and cannot enforce on $τ^2$-Bench, CAR-bench, and MedAgentBench, finding that 74\% of requirements are symbolically enforceable and 95\% of these need only simple, low-cost checks; and (3) an empirical evaluation of symbolic guardrails on the same three benchmarks, finding that symbolic guardrails improve security and safety without sacrificing utility, and often improve it. Our work draws attention to the potential for symbolic guardrails for AI agents, suggesting them as an overlooked but practical path toward deploying domain-specific AI agents in risk-averse commercial software. We release all codes and artifacts at https://github.com/hyn0027/agent-symbolic-guardrails.

AI安全符号推理领域代理形式化验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。