arXiv:2505.23020cs.CRcs.AI2025-05被引 10

用抽象行为链生成安全指令,让大模型在执行任务时更安全。

AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models

  • 用模拟环境生成真实可执行的指令,覆盖多步复杂动作。
  • 安全提升35.8%至79.5%,同时保持或提升模型帮助性。
  • 适合关注AI代理安全对齐的研究者与开发者。

大语言模型从知识提供者转向行动执行者,虽拓展了能力边界,但也显著增加了被恶意使用的风险。已有研究表明,当前基于LLM的代理在未被攻击时仍会执行大量恶意任务,表明后训练阶段存在安全对齐缺陷。为此,我们提出AgentAlign框架,利用抽象行为链作为安全对齐数据合成的媒介。通过在包含多种工具实例的模拟环境中实例化这些行为链,该框架能生成高度真实且可执行的指令,同时捕捉复杂的多步动态。为保障模型实用性,框架还通过非恶意解释按比例合成良性指令,精准校准有用性与无害性之间的边界。在AgentHarm上的评估显示,使用该方法微调三类开源模型,安全性能提升35.8%至79.5%,对帮助性影响极小甚至有所提升,优于多种提示方法。数据集与代码均已开源。

原文摘要 · Abstract (English)

The acquisition of agentic capabilities has transformed LLMs from "knowledge providers" to "action executors", a trend that while expanding LLMs' capability boundaries, significantly increases their susceptibility to malicious use. Previous work has shown that current LLM-based agents execute numerous malicious tasks even without being attacked, indicating a deficiency in agentic use safety alignment during the post-training phase. To address this gap, we propose AgentAlign, a novel framework that leverages abstract behavior chains as a medium for safety alignment data synthesis. By instantiating these behavior chains in simulated environments with diverse tool instances, our framework enables the generation of highly authentic and executable instructions while capturing complex multi-step dynamics. The framework further ensures model utility by proportionally synthesizing benign instructions through non-malicious interpretations of behavior chains, precisely calibrating the boundary between helpfulness and harmlessness. Evaluation results on AgentHarm demonstrate that fine-tuning three families of open-source models using our method substantially improves their safety (35.8% to 79.5% improvement) while minimally impacting or even positively enhancing their helpfulness, outperforming various prompting methods. The dataset and code have both been open-sourced.

安全对齐大模型代理行为链仿真生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。