将定位编辑方法移植到掩码扩散语言模型,发现编辑效果在长文本上显著下降。
Knowledge Editing in Masked Diffusion Language Models

- 采用定位-编辑框架,在双向建模的掩码扩散模型中定位并修改知识
- 单标记编辑在两类模型中均有效,但多标记编辑在扩散模型中性能大幅下降
- 提出针对中间生成状态的优化策略,显著提升长文本编辑成功率
知识编辑旨在更新或修正语言模型中的事实性知识。主流方法“定位-编辑”分为两步:先定位模型中的特定事实,再修改对应权重。现有方法仅适用于自回归模型(ARMs)。本文研究其是否适用于掩码扩散模型(MDMs),这类模型以双向建模和迭代去噪方式生成文本,而非逐词预测。我们对比了两种MDMs(LLaDA、Dream)与两种ARMs(LLaMA、Qwen)在相同规模下的表现。核心发现有两点:第一,编辑位置在两类模型中具有可转移性——因果追踪显示,两者均在最后主语标记处的早期至中期前馈层(MLP)最有效;第二,同一位置未带来一致效果:单标记编辑在两类模型中均成功,但随着目标长度增加,MDMs的编辑效果系统性下降,而ARMs保持稳定。失败原因在于生成多标记内容需经过部分未掩码的中间状态,而这些状态未被编辑优化。基于此诊断,我们提出一种简单修正方法,对中间状态进行优化,显著恢复了多标记编辑性能。
原文摘要 · Abstract (English)
Knowledge editing aims to update or correct factual knowledge in a language model. A widely used approach, locate-then-edit, does this in two steps: it first localizes a fact within the model, then edits the weights there. To date, such methods have been developed exclusively on autoregressive models (ARMs). Whether their underlying assumptions hold for masked diffusion models (MDMs), which model text bidirectionally and generate by iterative denoising rather than next-token prediction, remains an open question. We address it by transferring locate-then-edit to MDMs and comparing two MDMs (LLaDA, Dream) with two ARMs (LLaMA, Qwen) at matched scale. Our central finding has two parts. First, where an edit is applied transfers across paradigms: causal tracing highlights the same early-to-mid-layer MLP at the last subject token in both, and editing is most effective there. Second, this shared location does not guarantee a shared outcome. Single-token edits succeed in both, but as targets grow longer, editing degrades systematically in the MDMs but not the ARMs. The failure stems from how the edited fact is generated: producing a multi-token target requires passing through partially unmasked intermediate states for which the edit was never optimized. Guided by this diagnosis, we introduce a simple correction that optimizes the edit for these states, substantially restoring multi-token performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。