让有害知识为安全服务,动态路由实现更安全更智能的模型生成
Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routing

- 将有害数据转为专用专家模块,通过轻量门控网络动态调度
- 安全响应率提升超20%相对改进,有害但有益的问题回答更精准
- 零样本泛化强,无需额外标注即可适配新领域和任务
当前大语言模型对齐主流方法依赖于抹除——过滤有害数据或训练模型拒绝危险指令。尽管能有效降低即时毒性,却严重限制了模型的认知范围,导致系统过度谨慎,对敏感但无害的问题给出空洞拒绝。本文挑战这一教条,提出一种辩证对齐范式:有害数据蕴含丰富、特定领域的知识,对生成精准、安全、有信息量的内容至关重要。为此,我们提出SafeMoE框架,采用混合专家(MoE)结构,将有害知识隔离至特定领域的低秩适配器(LoRA专家)中,并仅在有害语料上训练。为从这些有害基础单元中合成安全输出,我们使用少量精心筛选的安全-信息型回复训练一个轻量级门控网络。推理时,该路由机制动态协调有害专家,有效引导生成轨迹,在利用其深度领域知识的同时严格遵守安全约束。大量实证评估显示,SafeMoE不仅更安全,安全响应率相对提升超过20%(绝对增益超15%),且在安全性与危害性并重时生成更具信息量的回答。此外,路由机制展现出强大的零样本泛化能力,无需领域特定监督即可扩展至未见领域与更广安全任务。研究揭示对齐范式应转变:真正的安全不在于屏蔽有害知识,而在于可控整合。
原文摘要 · Abstract (English)
The prevailing paradigm in large language model (LLM) alignment operates via erasure, filtering unsafe data or training models to strictly refuse harmful prompts. While effective at reducing immediate toxicity, this approach fundamentally constricts the model's epistemological scope, resulting in over-cautious systems that output uninformative blanket refusals to sensitive yet benign queries. In this work, we challenge the orthodoxy that unsafe data must be discarded. We propose a dialectical approach to alignment, positing that unsafe data encodes rich, domain specific knowledge critical for nuanced, safe, and informative generation. To operationalize this, we introduce SafeMoE, a Mixture-of-Experts (MoE) framework that isolates unsafe knowledge into domain-specific Low-Rank Adapters (LoRA experts) trained exclusively on harmful corpora. To synthesize safety from these unsafe primitives, we train a lightweight gating network using a minimal, highly curated set of safe-informative responses. During inference, this router dynamically orchestrates the unsafe experts, effectively steering the generation trajectory to harness their deep domain knowledge while strictly enforcing safety constraints. Extensive empirical evaluations across stringent safety benchmarks demonstrate that SafeMoE is not only safer, achieving over a 20% relative improvement in safe response rate (more than a 15% absolute gain), but also produces more informative responses when safety and harmfulness are of paramount concern. Furthermore, the routing mechanism exhibits strong zero-shot generalization to unseen domains and broader safety tasks without domain-specific supervision. Our findings suggest a paradigm shift in alignment: true safety requires not the masking of unsafe knowledge, but its controlled integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。