安全规则在智能体摘要中看似留存,实则失效,检测需超越文字存在。
AI Guardrail Survival under Single-Cycle Agentic Self-Summarization
- 摘要时规则可能变残余,形似实非,无法真正约束行为。
- 残余规则使违规行为频次提升34至57个百分点,远超完整规则。
- 仅查文字是否存在会误判安全,适合做安全审计的开发者必看。
长时间运行的智能体定期压缩上下文,用模型生成的摘要替代原始记录。近期研究发现,压缩过程中丢弃安全约束会导致多种模型行为失范(治理衰减;Chen, 2026)。本文聚焦单次压缩周期:安全规则如何丢失?这又对检测与评估意味着什么?核心发现是:存在性检查≠安全检查。即使规则未被直接删除,也常残留一个看似规则但无实际作用的残余物。在行为回放测试中,这种残余导致模型执行禁止动作的频率远高于完整规则(两种回放模型下分别提升+34和+57点,均为正向差异)。类别级生存表现类似残余现象,且完整规则有时也无法触发。因此,仅检查文本是否存在会带来虚假安全。进一步发现,规则形式条目比同等显著度的事实更易保留,这解释了为何基于存在的检查看似有效,实则无效。文本丢失具有场景依赖性(单一规则下为‘焊接或丢弃’;预算紧时出现谓词损失残余),未观察到预设的文本切断模式。此类损失运行时无声,只能通过与外部真实基准(如约束注册表)对比才能检测,可揭示文本缺失,但无法判断存活规则是否仍能生效。我们还发现评估陷阱:仅依赖LLM裁判标签可能导致结论反转。所有结果均基于单次压缩周期。
原文摘要 · Abstract (English)
Long-running agents periodically compact their context, replacing the transcript with a model-generated summary. Recent work shows that dropping a standing safety constraint during compaction drives behavioral violations across many models (Governance Decay; Chen, 2026). We ask a finer question: under a single compaction cycle, how is a safety rule lost, and what does that imply for detection and evaluation? Our central finding is that a presence check is not a safety check: when compaction does not drop a rule outright, it often leaves something that looks like a rule but does not act like one. On behavioral replay, a degraded residue leads the model to perform the prohibited action far more often than an intact welded rule does (all-case gaps of +34 and +57 points under two replay models, both positive), category-level survival behaves like a residue, and even intact rules sometimes fail to fire, so an audit that checks only textual presence gives false assurance. Sharpening this, rule-form items are retained substantially more often than prominence-matched facts, which is exactly why presence-based checking feels adequate even though survival is not protection. Textual loss is regime-dependent (weld-or-drop with a single rule; degraded predicate-loss residues under a tighter budget), and we did not observe the hypothesized textual severing mode. Such loss is silent at runtime and detectable only by comparison with retained external ground truth (such as a constraint registry), which reveals textual absence but not whether a surviving rule still fires. We also document evaluation pitfalls where LLM-judge labels alone would have reversed a conclusion. All results concern a single compaction cycle.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。