长对话中压缩上下文会悄悄删掉安全规则,导致大模型违规操作。
Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents
- 通过压缩上下文让安全规则消失,引发模型后期违规行为。
- 实验显示违规率从0%升至59%,规则被删时违规率达38%。
- 提出约束锁定技术,无需训练即可恢复安全,适合部署级模型
现代大模型代理依赖上下文压缩、摘要或淘汰来维持长会话的令牌预算。我们发现这一上下文管理层是关键的安全失效面:在上下文中可见的治理约束,代理会可靠遵守,但可能被压缩无声移除,导致代理在会话后期执行禁止的工具调用。我们将此现象称为治理衰减(Governance Decay)。我们引入ConstraintRot基准,包含可确定性评分的长周期代理场景,并测量七种模型族在压缩下的违规情况。在1,323个实验中,当策略完整保留在上下文中时违规率为0%,压缩后上升至30%,部分模型达59%;当约束幸存于摘要,违规仍为0%,若被丢弃则违规率达38%。我们进一步研究了压缩-驱逐攻击,即恶意内容引导摘要器忽略合法策略,结果显示优化注入能攻破所有评估模型。最后提出无需训练的约束锁定(Constraint Pinning)缓解方案,将违规率恢复至0%。这些结果揭示上下文管理是部署中大模型代理的一类核心治理表面。
原文摘要 · Abstract (English)
Modern LLM agents increasingly rely on context compaction, summarization, or eviction to keep long-running sessions within a token budget. We show that this context-management layer is a safety-critical failure surface: in-context governance constraints that agents reliably obey while visible can be silently removed by compaction, causing the same agent to perform prohibited tool actions later in the session. We call this failure mode Governance Decay. We introduce ConstraintRot, a benchmark of long-horizon agent scenarios with deterministic tool-call grading, and measure compaction-induced violations across seven model families. Across 1,323 episodes, violation rises from 0% with the policy in full context to 30% after compaction, reaching 59% for some models; when the constraint survives the summary, violation remains 0%, but when it is dropped, violation reaches 38%. We further study a Compaction-Eviction Attack, in which adversarial in-context content biases the summarizer to omit a legitimate policy, and show that optimized injections defeat every evaluated model. Finally, we propose Constraint Pinning, a simple training-free mitigation that quarantines governance constraints from lossy compaction and restores violation to 0% in our benchmark. These results identify context management as a first-class governance surface for deployed LLM agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。