大模型代理在对话初期最不安全，完成几轮常规任务后安全性显著提升。
The Cold-Start Safety Gap in LLM Agents
- 通过控制前置任务数，系统研究代理安全性的动态变化。
- 前置20个常规任务后，安全性能提升9%至52%。
- 建议部署前让代理先执行几轮常规任务以增强安全性。
工具调用型大模型代理在对话初期最易出错：我们发现其安全性随前置常规任务数量增加而显著提升，这一现象称为‘冷启动安全差距’。为此，我们提出SODA基准，可控制代理在遭遇安全威胁前完成最多20个常规任务。评估7个来自4个模型家族的代理，安全性能随前置任务从0增至20时提升9%–52%。表征分析显示，模型隐藏状态逐渐向安全对齐区域转移。进一步分析表明，前置任务本身是提升安全性的主要因素，而代理自身历史响应影响较小但对后续能力保持至关重要。在AgentHarm、Agent Safety Bench等开源安全基准以及BFCL、API-Bank等效用基准上均验证了该结论。因此，建议部署前让代理先完成若干常规任务以缓解冷启动风险。代码已开源。
原文摘要 · Abstract (English)
Are tool-calling LLM agents equally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a session and become substantially safer after a few regular agentic tasks -- a phenomenon we term the cold-start safety gap. To study this systematically, we introduce Safety Over Depth for Agents (SODA), a benchmark that controls how many regular agentic tasks the agent completes before encountering a safety threat, supporting up to 20 preceding tasks. Evaluating 7 models from 4 families, safety improves by 9--52% as the number of preceding regular agentic tasks increases from zero to twenty. Representation analysis confirms that model hidden states gradually shift toward a safety-aligned region as more preceding tasks are present. By systematically studying which part of the preceding conversation matters most, we find that the regular agentic tasks themselves are the primary driver of safety, while the agent's own prior responses have less effect on safety but are essential for preserving later utility. This conclusion is further supported by evaluation on open-source safety benchmarks (AgentHarm, Agent Safety Bench) and utility benchmarks (BFCL, API-Bank), confirming that warming up the agent with regular agentic tasks before deployment makes it safer and preserves full capability. Based on these findings, we recommend a simple deployment strategy: having the agent complete a few regular agentic tasks before possible exposure to safety-critical requests mitigates the cold-start safety gap. Our code is available at https://github.com/Trustworthy-ML-Lab/Agent-Cold-Start-Safety-Gap
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。