测试大模型在安全代理任务中，去安全限制后表现是否提升。
Measuring Safety Alignment Effects in Autonomous Security Agents
- 构建30个固定工具的漏洞分析任务，量化模型行为差异。
- 去安全限制后Gemma模型成功率最高达14.0%，拒绝率归零。
- 安全效果需系统评估，不能只看拒绝率，应分拆动作、工具、证据等维度。
现有单轮拒绝基准无法回答：当标准安全对齐语言模型及其去约束或删减版本作为自主安全代理运行时，行为是否不同？安全代理需在授权沙箱内检查代码库、调用工具并生成漏洞证据。本文提出一个基于追踪的基准，包含30个本地漏洞分析任务，固定工具、确定性成功判定、文本擦除规则和溯源验证。对比四个主流模型及其去约束/删减版本：Gemma 4 31B、Gemma 4 26B A4B、Qwen2.5-Coder 7B 和 Llama 3.1 8B。数据集包含1,500条安全代理轨迹与800条非安全控制轨迹。结果显示，去约束的Gemma 31B在安全任务上成功率从0.7%升至14.0%，均值溯源得分从3.27升至3.91（满分5），拒绝、异常动作及不安全动作发生率为0；而26B版本则从0.0%升至10.7%。但其他模型如Qwen2.5-Coder的去约束版本成功率反而从5.3%降至2.0%，且删减版Llama未遵守工具协议。所有模型在高难度触发证明与补丁验证任务上均未解决。结果表明，安全对齐效果必须在系统层面测量，需区分拒绝、不安全动作、工具可靠性与证据溯源，而非仅依赖拒绝率作为安全信号。
原文摘要 · Abstract (English)
Do stock safety-aligned language models and their uncensored or abliterated derivatives behave differently when run as autonomous security agents? Single-turn refusal benchmarks cannot answer this question: security agents must inspect repositories, call tools, and produce vulnerability evidence inside authorized sandboxes. We present a trace-based benchmark of 30 local vulnerability-analysis tasks with fixed tools, deterministic success predicates, redaction rules, and grounding checks, and compare four stock models against uncensored or abliterated derivatives: Gemma 4 31B, Gemma 4 26B A4B, Qwen2.5-Coder 7B, and Llama 3.1 8B. The artifact contains 1,500 security-agent traces and 800 non-security control traces. The Gemma pairs show large less-restricted gains on security tasks: 14.0% versus 0.7% success for 31B and 10.7% versus 0.0% for 26B, with higher mean grounding (3.91 versus 3.27 and 4.12 versus 1.64 out of five) and 0.0% refusal, suppressed-action, and unsafe-action rates in the 31B traces. However, controls and non-Gemma pairs rule out a clean security-specific or universal less-restricted effect: Gemma gaps also appear on ordinary coding tasks, Qwen2.5-Coder success is lower for the less-restricted derivative (2.0% versus 5.3%), and the abliterated Llama derivative fails the tool protocol. Across all families, hard proof-of-trigger and patch-verification tasks remain unsolved. These results show that safety alignment effects in autonomous security agents should be measured at the system level, separating refusal, unsafe action, tool reliability, and evidence grounding rather than treating refusal rate as the safety signal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。