arXiv:2505.18832cs.CV2025-05NeurIPS被引 5

定位扩散Transformer中知识的存储位置,实现精准模型编辑。

Localizing Knowledge in Diffusion Transformers

  • 提出通用方法,识别知识在DiT模块中的具体分布位置。
  • 在6类知识上验证,定位结果与生成内容有因果关联。
  • 适用于个性化和知识删除,效率高且不破坏原有能力。

理解生成模型中知识在各层的分布对提升可解释性、可控性和适应性至关重要。尽管已有研究探索了基于UNet架构的知识定位,基于扩散Transformer(DiT)的模型在此方面仍缺乏深入研究。本文提出一种模型和知识无关的方法,用于定位特定类型知识在DiT块中的编码位置。我们在PixArt-alpha、FLUX和SANA等前沿DiT模型上,针对六类不同知识进行了评估。结果表明,所识别出的模块具有可解释性,并与生成输出中的知识表达存在因果联系。基于此,我们将该框架应用于模型个性化与知识遗忘两个关键任务。在两种场景下,局部化微调均实现了高效、精准的更新,显著降低计算开销,提升任务性能,同时最小化对无关内容的影响,有效保留模型通用能力。总体而言,本研究揭示了DiT内部结构的新认知,为更可解释、高效、可控的模型编辑提供了实用路径。

原文摘要 · Abstract (English)

Understanding how knowledge is distributed across the layers of generative models is crucial for improving interpretability, controllability, and adaptation. While prior work has explored knowledge localization in UNet-based architectures, Diffusion Transformer (DiT)-based models remain underexplored in this context. In this paper, we propose a model- and knowledge-agnostic method to localize where specific types of knowledge are encoded within the DiT blocks. We evaluate our method on state-of-the-art DiT-based models, including PixArt-alpha, FLUX, and SANA, across six diverse knowledge categories. We show that the identified blocks are both interpretable and causally linked to the expression of knowledge in generated outputs. Building on these insights, we apply our localization framework to two key applications: model personalization and knowledge unlearning. In both settings, our localized fine-tuning approach enables efficient and targeted updates, reducing computational cost, improving task-specific performance, and better preserving general model behavior with minimal interference to unrelated or surrounding content. Overall, our findings offer new insights into the internal structure of DiTs and introduce a practical pathway for more interpretable, efficient, and controllable model editing.

扩散模型知识定位模型编辑可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。