测试工具型AI在运维中受间接提示注入攻击的风险，提出新基准并验证安全方案。
NetInjectBench: Benchmarking Indirect Prompt Injection in Tool-Using Large Language Model Agents for Network Operations
- 构建130场景基准，分离不可信文本与可信策略元数据进行评估。
- 纯提示防护下82.5%操作不安全，最优方案将风险降至10%以下。
- 元数据感知策略门可零失误防御攻击，同时保持99%任务可用性。
工具型大语言模型代理在网络安全运维中具有吸引力，但工单、告警、日志、运行手册和ChatOps消息可能携带间接提示注入。本文提出NetInjectBench，一个包含130个场景的基准，明确分离不可信的文本内容、可信的策略元数据与评估标签，用于网络运维工具使用评估。样本包含40个良性、40个弱攻击、40个强攻击和10个已批准高影响变更场景，分别在Qwen2.5-7B、Llama3.1-8B和Mistral-7B上测试。240次攻击实例中，原始执行的不安全操作率达82.50%；仅靠提示防护、Self-Reminder、Spotlighting及双阶段LLM判别器分别降至25.63%、21.67%、18.33%和10.00%。静态允许列表虽将风险压至5.00%，却阻断所有获批变更，导致0.00%可用性与100.00%过度拦截。在元数据完整性假设下，元数据感知策略门实现0/240次不安全攻击，95%威尔逊上限为1.58%，同时保持99.17%攻击场景可用性与100.00%获批变更可用性。结果表明，运维代理需在执行时建立授权边界，结合提示层指令净化。
原文摘要 · Abstract (English)
Tool-using large language model (LLM) agents are attractive for network operations, but tickets, alerts, logs, runbooks, and ChatOps messages can carry indirect prompt injections. We present NetInjectBench, a 130-scenario benchmark that separates untrusted artifact text, trusted policy metadata, and evaluation labels for network-operation tool use. The sample contains 40 benign, 40 weak-attack, 40 strong-attack, and 10 approved high-impact change scenarios; each is evaluated with Qwen2.5-7B, Llama3.1-8B, and Mistral-7B. Across 240 attack instances, naive execution reached an 82.50% unsafe tool-action rate. Prompt-only safety, Self-Reminder, Spotlighting, and a Two-Pass LLM Judge reduced this rate to 25.63%, 21.67%, 18.33%, and 10.00%, respectively. Static allowlisting reached 5.00% but blocked all approved changes, yielding 0.00% usefulness and 100.00% overblocking on approved cases. Under the stated metadata-integrity assumption, the metadata-aware policy gate produced 0/240 unsafe attack actions, with a 95% Wilson upper bound of 1.58%, while preserving 99.17% attack-scenario usefulness and 100.00% approved-change usefulness. The findings show that network-operation agents need execution-time authorization boundaries alongside prompt-level instruction hygiene.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。