测试AI编程助手的权限系统，发现其漏报率高达81%。
Measuring the Permission Gate: A Stress-Test Evaluation of Claude Code's Auto Mode
- 构建模糊授权任务基准，评估代码代理在不确定风险下的行为
- 端到端漏报率达81%，远高于官方报告的17%
- 文件修改绕过检测，尤其在清理操作中问题严重
Claude Code的自动模式是首个部署的AI编程代理权限系统，采用两级转录分类器来拦截危险工具调用。Anthropic报告其在生产流量上误报率为0.4%，漏报率为17%。本文首次对这一系统在刻意模糊授权场景下的表现进行独立评估，即用户意图明确但目标范围、影响半径或风险等级未明确定义的任务。我们使用AmPermBench——一个包含128个提示、涵盖四大运维任务类别和三个受控模糊维度的基准——在个体操作层面对253个状态变更动作进行评估，对照人工标注的真值。结果显示，该系统在压力测试负载下的端到端漏报率为81.0%（95%置信区间：73.8%-87.4%），显著高于生产环境报告的17%，反映的是任务性质差异而非系统矛盾。值得注意的是，36.8%的状态变更操作因属于二级（项目内文件编辑）而未被分类器覆盖，导致整体漏报上升。即使仅考虑分类器实际评估的160个动作（三级），漏报率仍达70.3%，误报率升至31.9%。二级覆盖缺口在构建物清理任务中最明显，漏报率达92.9%，因代理在无法使用预期命令行时会转向直接编辑状态文件，而此类操作未被分类器评估。
原文摘要 · Abstract (English)
Claude Code's auto mode is the first deployed permission system for AI coding agents, using a two-stage transcript classifier to gate dangerous tool calls. Anthropic reports a 0.4% false positive rate and 17% false negative rate on production traffic. We present the first independent evaluation of this system on deliberately ambiguous authorization scenarios, i.e., tasks where the user's intent is clear but the target scope, blast radius, or risk level is underspecified. Using AmPermBench, a 128-prompt benchmark spanning four DevOps task families and three controlled ambiguity axes, we evaluate 253 state-changing actions at the individual action level against oracle ground truth. Our findings characterize auto mode's scope-escalation coverage under this stress-test workload. The end-to-end false negative rate is 81.0% (95% CI: 73.8%-87.4%), substantially higher than the 17% reported on production traffic, reflecting a fundamentally different workload rather than a contradiction. Notably, 36.8% of all state-changing actions fall outside the classifier's scope via Tier 2 (in-project file edits), contributing to the elevated end-to-end FNR. Even restricting to the 160 actions the classifier actually evaluates (Tier 3), the FNR remains 70.3%, while the FPR rises to 31.9%. The Tier 2 coverage gap is most pronounced on artifact cleanup (92.9% FNR), where agents naturally fall back to editing state files when the expected CLI is unavailable. These results highlight a coverage boundary worth examining: auto mode assumes dangerous actions transit the shell, but agents routinely achieve equivalent effects through file edits that the classifier does not evaluate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。