后置安全训练易被攻破,因它只是掩盖而非消除模型能力。
The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists

- 安全更新在能力方向上正交,仅抑制少数高曲率方向。
- 100步良性微调即恢复拒绝能力,跨5类模型复现该现象。
- 预训练阶段早期植入安全信号,才能持久抵抗攻击。
后置安全训练(如RLHF、DPO)是主流对齐方法,但劫持攻击、微调攻击和激活空间探测仍能恢复被压制行为。本文从几何角度解释其脆弱性:安全更新Δ = W_safe - W_base 与模型能力曲率(基于能力损失的实证费雪信息)几乎正交,且其子空间内部分集中于少数高曲率方向。该更新呈薄而锐利特征,如同在完整能力上叠加拒绝门,而非彻底消除。核不可移动引理说明此类更新只能掩蔽能力,无法移除,轻微良性微调即可恢复——100步微调使Qwen-2.5-7B与Llama-3-8B Instruct的拒绝失效,且跨五类模型复现。对OLMo-2-1B的267个检查点分析显示,安全基底在约60亿至600亿预训练词元间发生突变式涌现。进一步构造性验证:从头训练中联合训练安全性的模型,在410M至6.9B规模下,拒绝率达87%~98%,攻击后仍保持84%~91%,侵蚀仅2~14个百分点;而计算量匹配的窗口调度法无法建立持久拒绝。安全信号在预训练中的持续存在,而非其时机,才是抗攻击的关键。
原文摘要 · Abstract (English)
Post-hoc safety training (RLHF, DPO) is the dominant way to align large language models, yet jailbreaks (Zou et al., 2023b), fine-tuning attacks (Qi et al., 2024), and activation-space probes (Arditi et al., 2024) keep recovering the behaviors it was meant to remove. We give this fragility one geometric explanation and trace it to when, during pretraining, safety can take hold. We measure the safety update $\Delta = W_{\text{safe}} - W_{\text{base}}$ against the curvature of the model's capabilities (the empirical Fisher of a capability loss). Post-hoc safety consistently lands in a suppression regime: $\Delta$ is nearly orthogonal to the capability directions, and its small in-subspace part concentrates on a few high-curvature ones. The update is thin but sharp, a refusal gate laid over intact capabilities rather than erasure of them. A kernel-immobility lemma explains why such an update can only mask a capability, not remove it, so a little benign fine-tuning restores it: 100 steps of benign fine-tuning collapse refusal on Qwen-2.5-7B and Llama-3-8B Instruct at preserved capability, a signature that replicates across five model families. Following the account into pretraining, a 267-checkpoint sweep of OLMo-2-1B (OLMo et al., 2025) shows the substrate that safety engages emerging in a sharp transition between roughly 6B and 60B pretraining tokens. We then use the account constructively: models trained from scratch with safety co-training spread continuously across pretraining reach 87 to 98% refusal whose post-attack level holds at 84 to 91% at every scale, an erosion of 2 to 14 pp against 35 to 38 pp for post-hoc installs, at capability matched or better than an LM-only baseline and holding from 410M to 6.9B, whereas a compute-matched windowed schedule installs no lasting refusal. Persistence of the safety signal across pretraining, not its timing, is what buys attack robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。