在真实企业安全限制下评估12个编程智能体,发现安全策略会显著降低性能和效率。
Permission Denied: Policy-Graded Evaluation of Coding Agents in Hardened Environments
- 构建多层安全策略测试环境,模拟企业真实限制条件。
- 最严策略下成功率下降18.3分,成本飙升167.3%,模型表现差异大。
- 开源边界测试工具,支持在受限环境下评估编程智能体。
编程智能体在组织中运行时,常受权限限制、网络出口管控、只读文件系统及非根执行等安全策略约束,类似普通软件。然而现有基准大多在宽松沙盒中评估,无法反映策略下的真实表现。本文在终端基准Terminal-Bench 2.1上,对12个编程智能体在多层次嵌套安全策略下进行评估,策略源自常见企业实际限制。结果显示,安全加固并非无代价,且影响不均:在最严格策略下,成功率最多下降18.3分,成本增加167.3%;成功与效率表现存在矛盾,最优成功率模型反而最耗时。此外,失败模式表现为超时或错误解,而非提前终止,且不同模型表现各异。为确保公平比较,验证了任务在最严策略下的可解性,区分模型缺陷与策略导致的不可行。研究发布Boundary-Bench,一个开源插件,支持在Terminal-Bench及兼容基准上进行策略约束下的智能体评估。
原文摘要 · Abstract (English)
Coding agents increasingly run inside organizations whose security controls (scoped credentials, restricted egress, read-only filesystems, non-root execution) constrain them like any other software. Existing benchmarks, however, evaluate agents almost exclusively in permissive sandboxes, so it is unknown how performance changes when policy is enforced. In this work, we evaluate 12 coding agents on Terminal-Bench 2.1 across nested security policy levels derived from common real-world enterprise restrictions. Hardening is never free but far from uniform: under the strictest policy, success losses reach 18.3 points and cost inflation 167.3\%, and the two axes disagree; the model that best preserves success is also the one that loses the most efficiency, so model choice is policy-dependent. Beyond aggregate scores, we characterize how agents behave when policy blocks their actions and decompose the failures hardening induces: runs grind into timeouts or wrong solutions rather than stopping early, in a mix that differs by model. To ground comparisons, we verify task solvability under the strictest policy, separating model failures from tasks the policy forecloses. We release Boundary-Bench, an open-source hardening plugin enabling policy-constrained evaluation of coding agents on Terminal-Bench and compatible benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。