用魔法令牌实现大模型安全行为的实时切换,高效可控。
Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Training
- 单阶段微调融合三种安全行为,通过令牌动态切换。
- 8B模型安全性能超越671B的DeepSeek-R1,训练成本大幅降低。
- 适合需要灵活部署安全策略的场景,如红队测试与内容过滤。
当前大型语言模型的内容安全方法(如SFT、RLHF)通常依赖多阶段训练流程,缺乏部署后的细粒度控制能力。为此,我们提出一种统一的共训练框架,在单一SFT阶段内高效集成正向(合法/亲社会)、负向(无过滤/高风险)和拒答型(拒绝导向/保守)三种安全行为。每种行为可通过简单的系统指令或魔法令牌在推理时动态激活,实现隐蔽且高效的运行时行为切换。该机制支持多样化的应用场景:正向用于安全用户交互,负向用于内部红队测试,拒答型则由上游内容审核信号触发。共训练策略在输出空间中诱导出显著分离的安全对齐边界,实证验证了模型的安全鲁棒性,并实现了前所未有的细粒度控制。实验表明,本方法在安全对齐质量上媲美SFT+DPO,其8B模型的安全性能甚至超过671B的DeepSeek-R1,同时显著降低训练复杂度与部署成本。本工作为大模型内容安全提供了可扩展、高效且高度可控的解决方案。
原文摘要 · Abstract (English)
Current methods for content safety in Large Language Models (LLMs), such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often rely on multi-stage training pipelines and lack fine-grained, post-deployment controllability. To address these limitations, we propose a unified co-training framework that efficiently integrates multiple safety behaviors: positive (lawful/prosocial), negative (unfiltered/risk-prone) and rejective (refusal-oriented/conservative) within a single SFT stage. Notably, each behavior is dynamically activated via a simple system-level instruction, or magic token, enabling stealthy and efficient behavioral switching at inference time. This flexibility supports diverse deployment scenarios, such as positive for safe user interaction, negative for internal red-teaming, and rejective for context-aware refusals triggered by upstream moderation signals. This co-training strategy induces a distinct Safety Alignment Margin in the output space, characterized by well-separated response distributions corresponding to each safety mode. The existence of this margin provides empirical evidence for the model's safety robustness and enables unprecedented fine-grained control. Experiments show that our method matches the safety alignment quality of SFT+DPO, with our 8B model notably surpassing DeepSeek-R1 (671B) in safety performance, while significantly reducing both training complexity and deployment costs. This work presents a scalable, efficient, and highly controllable solution for LLM content safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。