arXiv:2608.27839cs.AI2026-08

KLOD通过保留非目标分布,有效缓解知识编辑中的局部性退化问题。

KLOD: Locality-Preserving Knowledge Editing via Non-Target Distribution Preservation

  • 引入受限且分布保持的目标函数,分离目标更新与稳定分布
  • 在Llama3和Qwen模型上显著减少局部性退化,目标概率阈值可控
  • 适合需要高可靠性和长期稳定性的知识编辑场景

基于微调的知识编辑方法简单且与架构无关,但标准交叉熵会提升目标概率,而未显式约束非目标输出分布的变化。在序列编辑中,这种无约束的分布重分配会导致分布偏移累积,进而引发局部性退化。本文提出KLOD,一种有界且分布保持的微调目标,将预期的目标更新与应保持稳定的分布分离。KLOD在达到概率阈值后停止目标放大,同时在目标位置保持排除目标的非目标分布,在前缀位置保持完整的下一个词分布。在CounterFact和ZsRE数据集上使用Llama3-8B-Instruct和Qwen2.5-7B-Instruct的实验表明,KLOD显著缓解了局部性退化,同时保持高编辑可靠性。目标概率阈值还提供了可调控的泛化-局部性权衡。消融实验、多种子测试和分布KL分析支持这一结论:KLOD的局部性提升源于对输出分布的保持,而非单纯弱化编辑。代码已开源于GitHub:https://github.com/Hostoday/KLOD。

原文摘要 · Abstract (English)

Fine-tuning-based knowledge editing is simple and architecture-agnostic, but standard cross-entropy increases the edited target probability without explicitly constraining changes in the non-target output distribution. In sequential editing, such unconstrained redistribution can accumulate as distributional drift and contribute to locality degradation. We propose KLOD, a bounded and distribution-preserving objective for fine-tuning-based knowledge editing that separates the intended target update from distributions that should remain stable. KLOD stops target amplification once a probability threshold is reached, while preserving the target-excluded non-target distribution at target positions and the full next-token distribution at prefix positions. Experiments on CounterFact and ZsRE with Llama3-8B-Instruct and Qwen2.5-7B-Instruct show that KLOD substantially mitigates locality degradation while maintaining high edit reliability. The target probability threshold further provides a controllable Generalization--Locality trade-off. Ablation, multi-seed, and distributional KL analyses support the interpretation that KLOD's locality gains are associated with preserving output distributions rather than simply weakening the edit. Code is available on GitHub https://github.com/Hostoday/KLOD .

知识编辑微调优化分布保持局部性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。