通过失败轨迹实现智能体自进化，提升安全与效率的平衡。
On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment
- 用失败轨迹生成修复监督信号，无需人工标注。
- 在多个数据集上降低攻击成功率33.5%，有害服从率下降82.6%。
- 适合追求高安全性的自主智能体研发者使用。
使用工具的LLM智能体的失败体现在行为轨迹中，而非仅最终响应：可能执行不安全的工具调用、遵循注入指令、响应有害请求，或过度拒绝正常任务，尽管输出看似安全。现有安全对齐信号多为响应级或离策略，常导致安全与性能权衡。为此，我们提出FATE——一种基于失败轨迹的在策略自演化框架，将验证器评分的失败转化为修复监督，无需专家示范。每个失败由同一策略生成修复候选，经验证器重新评分并过滤安全、效用、过度拒绝控制及轨迹有效性后，形成密集的轨迹级监督信号，用于智能体自演化。过程中引入帕累托前沿策略优化（PFPO），结合监督预热与帕累托感知优化，保留安全-效用权衡。在AgentDojo、AgentHarm和ATBench上的实验表明，FATE在不同模型和规模下均提升安全性，同时保持有效行为。相比强基线，其攻击成功率降低33.5%，有害服从率下降82.6%，外部轨迹安全诊断能力提升6.5%。结果表明，失败轨迹可提供结构化修复监督，推动更安全的自演化智能体发展。
原文摘要 · Abstract (English)
Tool-using LLM agents fail through trajectories rather than only final responses, as they may execute unsafe tool calls, follow injected instructions, comply with harmful requests, or over-refuse benign tasks despite producing a seemingly safe answer. Existing safety-alignment signals are largely response-level or off-policy, and often incur a safety-utility trade-off: improving agent safety comes at the cost of degraded task performance. Such sparse and single-objective rewards severely limit real-world usability. To bridge this gap, we propose FATE, an on-policy self-evolving framework that transforms verifier-scored failures into repair supervision without expert demonstrations. For each failure, the same policy proposes repair candidates, which are then re-scored by verifiers and filtered across security, utility, over-refusal control, and trajectory validity. This dense trajectory-level information is then used as a supervision signal for agent self-evolution. During this process, we further introduce Pareto-Front Policy Optimization (PFPO), combining supervised warmup with Pareto-aware policy optimization to preserve safety-utility trade-offs. Experiments on AgentDojo, AgentHarm, and ATBench show that FATE improves safety across different models and scales while preserving useful behavior. Compared with strong baselines, FATE reduces attack success rate by 33.5%, harmful compliance by 82.6%, and improves external trajectory-safety diagnosis by 6.5%. These results suggest that failed trajectories can provide structured repair supervision for safer self-evolving agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。