arXiv:2606.12841cs.LGcs.AI2026-06

首个无需训练的扩散语言模型知识编辑框架,高效精准修改模型记忆。

TimeROME-DLM: Temporal Causal Tracing and Low-Rank Inference-Time Knowledge Editing for Masked Diffusion Language Models

论文配图:TimeROME-DLM: Temporal Causal Tracing and Low-Rank Inference-Time Knowledge Editing for Masked Diffusion Language Models
图 1 · 摘自论文原文
  • 通过时间因果追踪定位关键坐标,实现推理时精准干预
  • 单次正则化更新即可编辑400条知识,保持原输出稳定在1纳特内
  • 零额外显存消耗,速度比传统方法快4-14倍,适合大规模知识管理

掩码扩散语言模型(MDLMs)如LLaDA已媲美自回归大模型,但现有知识编辑与遗忘方法(如ROME、MEMIT)均针对自回归变换器设计,在迭代去噪过程中失效或需梯度更新,导致数十GB额外显存开销并破坏MDLM性能。本文提出TimeROME-DLM,首个无需训练、无梯度、推理时的知识编辑框架。其核心为时间间接效应(TIE)因果追踪协议,识别每个事实在后续去噪步骤中对目标预测影响最大的坐标;结合闭式低秩残差编辑内存,聚合所有遗忘事实的主体键与目标增量,仅在每次扩散前执行一次岭正则化更新,并通过稀疏化控制效用溢出。主干权重冻结,仅需调节三个超参数(alpha, lambda, q),在TOFU-finetuned LLaDA-8B-Base上使遗忘集对数似然降低约83 nats。同一配置可迁移至LLaDA-8B-Instruct、Dream-7B、MMaDA-8B、DiffuLLaMA-7B和LLaDA-MoE-1.4B。在50条连续插入的事实下,保留集对数似然变化小于1 nat,相比最强训练时基线实现4-14倍的壁钟加速,且显存零增加,支持亚线性扩展至400条知识。TimeROME-DLM以极低成本弥合了自回归模型与扩散模型在定位-编辑环节的差距。

原文摘要 · Abstract (English)

Masked diffusion language models (MDLMs) such as LLaDA now rival autoregressive (AR) LLMs, but every existing knowledge-editing and unlearning method (ROME, MEMIT, etc.) targets AR transformers and either makes assumptions that fail under iterative denoising, or requires gradient updates whose backward-pass activations cost tens of GB of extra VRAM and which collapse MDLMs at standard learning rates. We introduce TimeROME-DLM, the first training-free, gradient-free, inference-time knowledge-editing framework for MDLMs. It couples two components: a Temporal Indirect Effect (TIE) causal-tracing protocol that identifies, for each fact, the coordinate whose intervention most strongly drives the object prediction at later denoising steps; and a closed-form, low-rank residual edit memory that aggregates subject keys and target deltas across all forget facts and applies a single ridge-regularised update at that coordinate at every diffusion forward, with sparsification to limit utility spillover. Backbone weights stay frozen; only three hyperparameters (alpha, lambda, q) are tuned on a small validation split. On TOFU forget01 with TOFU-finetuned LLaDA-8B-Base, TimeROME-DLM cuts forget-set log-probability by roughly 83 nats. The same configuration transfers to LLaDA-8B-Instruct, Dream-7B, MMaDA-8B, DiffuLLaMA-7B, and LLaDA-MoE-1.4B. It keeps retain-set log-probability nearly flat (within ~1 nat at the utility-safe operating point) across 50 sequentially inserted facts, delivers a four- to fourteen-fold wall-clock speedup with zero additional VRAM over the strongest converged training-time baseline, and scales sub-linearly to 400 facts. TimeROME-DLM closes the locate-then-edit gap between AR LLMs and MDLMs at a fraction of the computational cost.

知识编辑扩散模型推理优化低秩更新

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。