通过激活转向实现扩散语言模型的高效推理控制,提升安全拒答效果。
Activation Steering for Masked Diffusion Language Models
- 从对比提示中提取低维激活方向,全局干预反向扩散过程。
- 在多模型上实现显著且系统的行为改变,优于提示和优化基线。
- 早期去噪步骤与中后期层是控制关键,跨语言迁移有效但不通用到自回归模型。
掩码扩散语言模型(MDLMs)通过迭代掩码词去噪生成文本,支持掩码并行解码,并在可控性与效率之间表现出异于自回归大模型的特点。然而,面向推理阶段的高效表示级控制机制仍待探索。为此,本文提出一种针对MDLMs的激活转向原语:仅需一次前向传播,从对比提示集中提取单一低维方向,并对反向扩散过程中残差流激活进行全局干预,无需优化或修改采样流程。以安全拒答为实际应用场景,发现多个MDLM中的拒答行为由约一维的激活子空间决定。应用对应方向可引发显著且系统的行为转变,效果远超提示和优化基线。进一步发现,有效方向不仅可从指令后标记提取,也可从通常在自回归模型中无效的指令前标记中获取,体现扩散特异性。消融实验表明,最大控制力集中于早期去噪步骤与中晚期变压器层,早期扩散块贡献尤为突出。此外,在英汉双语训练的MDLM中,提取方向在两语言间强迁移,但无法可靠泛化至自回归架构,凸显安全约束表征的架构依赖性。
原文摘要 · Abstract (English)
Masked diffusion language models (MDLMs) generate text via iterative masked-token denoising, enabling mask-parallel decoding and distinct controllability and efficiency tradeoffs from autoregressive LLMs. Yet, efficient representation-level mechanisms for inference-time control in MDLMs remain largely unexplored. To address this gap, we introduce an activation steering primitive for MDLMs: we extract a single low-dimensional direction from contrastive prompt sets using one prompt-only forward pass, and apply a global intervention on residual-stream activations throughout reverse diffusion, without performing optimization or altering the diffusion sampling procedure. Using safety refusal as a deployment-relevant case study, we find that refusal behavior in multiple MDLMs is governed by a consistent, approximately one-dimensional activation subspace. Applying the corresponding direction yields large and systematic behavioral shifts and is substantially more effective than prompt-based and optimization-based baselines. We further uncover diffusion-specific accessibility: effective directions can be extracted not only from post-instruction tokens, but also from pre-instruction tokens that are typically ineffective in autoregressive models due to causal attention. Ablations localize maximal leverage to early denoising steps and mid-to-late transformer layers, with early diffusion blocks contributing disproportionately. Finally, in an MDLM trained on English and Chinese, extracted directions transfer strongly between English and Chinese, but do not reliably generalize to an autoregressive architecture, highlighting architecture-dependent representations of safety constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。