针对扩散语言模型的安全部署位置,提出聚焦中间生成阶段的新对齐方法。
Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Position
- 发现中间令牌比开头更关键,是安全对齐的重点。
- 新方法MOSA在两个基准上抵御八种攻击,效果显著优于基线。
- 适合关注非自回归模型安全性的研究者和实践者。
扩散大语言模型(dLLMs)因其独特的训练与推理方式,已成为一种有竞争力的非自回归范式。然而,该新型架构的安全性研究尚属空白。本文首次分析dLLMs的安全表现,并提出一种针对其生成特性的新型安全对齐方法。我们发现防御方与攻击方存在关键不对称:防御方发现,响应中的中间令牌比初始令牌更影响整体安全性,提示对齐中间部分更有效;而攻击方则受限于实际中模型强烈的顺序生成倾向,难以操控中间令牌,被迫遵循生成分布。基于此不对称性,我们提出中令牌安全对齐(MOSA),通过强化学习直接对齐模型中间生成与安全拒绝。我们在两个基准上实现MOSA,并与八种攻击方法对比。结果表明MOSA在安全性上显著优于现有方法,且在编码、数学和通用推理任务中保持良好实用性。
原文摘要 · Abstract (English)
Diffusion Large Language Models (dLLMs) have recently emerged as a competitive non-autoregressive paradigm due to their unique training and inference approach. However, there is currently a lack of safety study on this novel architecture. In this paper, we present the first analysis of dLLMs' safety performance and propose a novel safety alignment method tailored to their unique generation characteristics. Specifically, we identify a critical asymmetry between the defender and attacker in terms of security. For the defender, we reveal that the middle tokens of the response, rather than the initial ones, are more critical to the overall safety of dLLM outputs; this seems to suggest that aligning middle tokens can be more beneficial to the defender. The attacker, on the contrary, may have limited power to manipulate middle tokens, as we find dLLMs have a strong tendency towards a sequential generation order in practice, forcing the attack to meet this distribution and diverting it from influencing the critical middle tokens. Building on this asymmetry, we introduce Middle-tOken Safety Alignment (MOSA), a novel method that directly aligns the model's middle generation with safe refusals exploiting reinforcement learning. We implement MOSA and compare its security performance against eight attack methods on two benchmarks. We also test the utility of MOSA-aligned dLLM on coding, math, and general reasoning. The results strongly prove the superiority of MOSA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。