arXiv:2510.09781cs.LGcs.AI2025-10被引 25

用合成数据训练前置安全防护模型,提前拦截智能体的危险计划。

Building a Foundational Guardrail for General Agentic Systems via Synthetic Data

  • 通过合成数据生成带风险标签的计划轨迹,构建预执行安全数据集。
  • 提出Safiron模型,在多个场景下实现高精度风险检测与类型分类。
  • 发布真实场景基准Pre-Exec Bench,支持多工具、多分支路径评估。

尽管大语言模型智能体能够规划多步任务,但在执行前干预计划阶段是防止危害最安全的方式,因为某些风险一旦执行便可能造成严重后果。然而现有防护机制多为事后检测,难以扩展且缺乏对计划层面的可控监督。为此,本文指出当前研究存在数据、模型和评估三大空白。为填补数据空白,提出AuraGen,一种可控制的生成引擎,能合成良性轨迹、注入带类别标签且难度可控的风险,并通过自动化奖励模型过滤,生成大规模可靠数据集用于预执行安全训练。为解决模型空白,提出基础防护模型Safiron,结合跨规划器适配器与紧凑型防护模型,统一输入格式,识别风险、标注类型并生成推理理由;采用两阶段训练策略,在广泛探索的数据配方下实现跨场景强泛化能力。为弥合评估空白,发布Pre-Exec Bench,一个覆盖多种工具和分支路径的真实基准,量化评估检测、细粒度分类、解释生成及跨规划器泛化能力,所有测试均经人工验证。大量实验表明,所提防护系统在Pre-Exec Bench上持续优于强基线,消融实验进一步提炼出可落地的安全实践模板。

原文摘要 · Abstract (English)

While LLM agents can plan multi-step tasks, intervening at the planning stage-before any action is executed-is often the safest way to prevent harm, since certain risks can lead to severe consequences once carried out. However, existing guardrails mostly operate post-execution, which is difficult to scale and leaves little room for controllable supervision at the plan level. To address this challenge, we highlight three critical gaps in current research: data gap, model gap, and evaluation gap. To close the data gap, we introduce AuraGen, a controllable engine that (i) synthesizes benign trajectories, (ii) injects category-labeled risks with calibrated difficulty, and (iii) filters outputs via an automated reward model, producing large and reliable corpora for pre-execution safety. To close the guardian model gap, we propose a foundational guardrail Safiron, combining a cross-planner adapter with a compact guardian model. The adapter unifies different input formats, while Safiron flags risky cases, assigns risk types, and generates rationales; trained in two stages with a broadly explored data recipe, Safiron achieves robust transfer across settings. To close the evaluation gap, we release Pre-Exec Bench, a realistic benchmark covering diverse tools and branching trajectories, which measures detection, fine-grained categorization, explanation, and cross-planner generalization in human-verified scenarios. Extensive experiments demonstrate consistent gains of the proposed guardrail over strong baselines on Pre-Exec Bench, and ablations further distill actionable practices, providing a practical template for safer agentic systems.

智能体安全合成数据风险检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。