安全分类器在嵌入漂移下会突然失效，即使模型变化很小也导致误判率飙升。
I Can't Believe It's Not Robust: Catastrophic Collapse of Safety Classifiers under Embedding Drift
- 用微小的嵌入扰动测试分类器鲁棒性，仅0.02幅度就引发性能崩溃
- 分类器准确率从85%暴跌至50%，但置信度仅下降14%，难以察觉异常
- 指令微调模型比基础模型更难区分类别，反而更难保障安全
指令微调推理模型常搭配基于冻结嵌入训练的安全分类器，假设表示稳定性。我们系统检验该假设，发现其失效：大小为σ=0.02的归一化扰动（相当于嵌入球面上约1°的角漂移），使分类器性能从85%的ROC-AUC降至50%。关键的是，平均置信度仅下降14%，导致72%的误分类发生在高置信度情况下，造成危险的无声失效，破坏常规监控机制。进一步发现，指令微调模型的类别可分性比基础模型差20%，使对齐系统反而更难防护。研究揭示了生产级AI安全架构的根本脆弱性，挑战了安全机制跨版本迁移的普遍假设。
原文摘要 · Abstract (English)
Instruction tuned reasoning models are increasingly deployed with safety classifiers trained on frozen embeddings, assuming representation stability across model updates. We systematically investigate this assumption and find it fails: normalized perturbations of magnitude $σ=0.02$ (corresponding to $\approx 1^\circ$ angular drift on the embedding sphere) reduce classifier performance from $85\%$ to $50\%$ ROC-AUC. Critically, mean confidence only drops $14\%$, producing dangerous silent failures where $72\%$ of misclassifications occur with high confidence, defeating standard monitoring. We further show that instruction-tuned models exhibit 20$\%$ worse class separability than base models, making aligned systems paradoxically harder to safeguard. Our findings expose a fundamental fragility in production AI safety architectures and challenge the assumption that safety mechanisms transfer across model versions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。