让大模型代理在安全与可用间平衡,通过反馈引导修正计划。
From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agents

- 用三类决策+自然语言反馈指导代理,动态调整任务计划。
- 在ASB和AgentHarm数据集上攻击成功率降至10.42%,效果最佳。
- 适合需要高安全性且不牺牲有用功能的智能体系统开发者。
基于大模型的防护机制通常在执行前评估动作或输入,生成如允许/拒绝、风险等级或违规解释等安全信号。然而,代理风险常源于良性任务被不可信外部内容、危险指令或高危工具使用污染。现有防护机制往往将整个任务整体标记为不安全,虽阻断威胁但牺牲了原有有益部分。此外,现有工作多孤立评估防护机制,未明确其干预是否真正提升代理下游行为的安全性。为此,我们提出TRIAD(三重响应迭代代理防护框架),将防护生成的语义反馈作为引导信号,在每个规划步骤中保持代理与良性目标对齐。我们在自构建训练数据集上微调语言模型,使其输出三种决策之一:继续、拒绝或更新,并附结构化自然语言反馈。不同于仅允许或阻止执行,‘更新’引导代理修改计划,规避有害成分,尽可能保留良性任务。TRIAD将此反馈注入代理上下文,实现防护反馈与规划的闭环。在ASB和AgentHarm上的大量实验表明,TRIAD将平均攻击成功率降至10.42%,在整合防护基线中实现最优安全-效用权衡。代码已开源:https://github.com/YUHAOSUNABC/TRIAD。
原文摘要 · Abstract (English)
LLM-based guardrails typically safeguard agents by evaluating proposed actions or inputs before execution, producing safety signals such as binary allow/deny decisions, risk categories, and/or explanatory rationales about potential policy violations. However, agent risks often arise when otherwise benign tasks are contaminated by untrusted external content, unsafe instructions, or risky tool use. Existing guardrails often flag the entire task uniformly as unsafe, thereby blocking the threat but sacrificing the benign part. Moreover, existing work largely evaluates guardrails in isolation, leaving unclear whether their interventions lead to safer downstream agent behavior. To address this, we introduce TRIAD (Tripartite Response for Iterative Agent Guardrailing), a guardrail-integrated agent framework that leverages guardrail-generated verbal feedback as a guiding signal to keep the agent aligned with benign objectives at each planning step. We finetune a language model on a self-curated training dataset to output one of three decisions: proceed, refuse, or update, together with structured natural-language feedback. Rather than merely allowing or blocking execution, update guides the agent to revise its plan, avoid harmful components, and preserve the benign task where possible. TRIAD injects this feedback into the agent's context, enabling subsequent plan revision and forming a closed loop between guardrail feedback and agent planning. Extensive experiments on ASB and AgentHarm show that TRIAD reduces the average attack success rate to 10.42%, while achieving the best safety-utility trade-off among guardrail-integrated baselines. Our code is available at: https://github.com/YUHAOSUNABC/TRIAD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。