通过锚定安全电路,让大模型进化更安全。
Safe Evolution with Circuit Anchors
- 识别并固定模型中不到2%的关键安全电路,其余部分自由演化。
- 实验显示新方法在3个模型族上显著提升安全性,能力损失极小。
- 适合关注大模型安全演化的研究者与开发者使用。
在生物进化中,无约束突变可能导致灾难性后果:生物体可能增强能力却丧失生存必需功能。自然的解决方案是发育约束,即核心调控基因保持稳定,而外围基因自由适应。我们发现当前大语言模型的自进化算法缺乏类似约束,仅追求能力提升,隐含假设安全会自动保留。实验表明这一假设极为危险:模型可能‘误进化’成强大但危险的实体。受5亿年演化中Hox基因维持身体结构的启发,我们提出电路锚定进化(CAE)。通过机制可解释性,我们识别出一个不足2%模型特征组成的安全电路,其因果决定安全行为。在进化过程中锚定该电路,限制其偏移范围,同时允许其余特征自由演化。这模仿了‘有约束的可进化性’原则:保留核心,适应外围。在3个模型家族和两种进化算法上的实验表明,CAE在安全性保持方面显著优于基于显式奖励的约束,在效果与效率上均表现更优。正如发育约束防止生物产生不可存活的个体,电路锚定也防止模型演化出强大但危险的系统。
原文摘要 · Abstract (English)
In biological evolution, unconstrained mutation can lead to catastrophic outcomes: organisms may evolve enhanced capabilities while losing essential functions for survival. Nature's solution is \textit{developmental constraints}, where core regulatory genes remain anchored while peripheral genes adapt freely. We observe that current self-evolution algorithms for large language models lack analogous constraints. They optimize purely for capability, implicitly assuming safety will be preserved. Our experiments reveal this assumption to be dangerously wrong: models can \textit{misevolve} into powerful yet dangerous entities. Inspired by how Hox genes anchor body structure across $500$ million years of evolution, we propose \textbf{Circuit-Anchored Evolution (CAE)}. Using mechanistic interpretability, we identify a tiny \textit{safety circuit}, comprising less than $2$\% of model features, that causally mediates safety behaviors. We anchor this circuit during evolution, constraining it within a small displacement bound while allowing the remaining features to evolve freely. This mirrors the biological principle of \textit{evolvability with constraint}: preserving what is essential while adapting what is peripheral. Experiments across $3$ model families and two evolution algorithms demonstrate that CAE achieves superior safety preservation with minimal capability loss, substantially outperforming explicit reward-based constraints in both effectiveness and efficiency. Just as developmental constraints prevent biological evolution from producing nonviable organisms, circuit anchoring prevents model evolution from producing capable but dangerous systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。