提出双模态编辑方法,高效更新视觉语言模型知识
DualEdit: Dual Editing for Knowledge Updating in Vision-Language Models
- 在不同层同时编辑文本与视觉模态,提升更新效率
- 新知识更新成功率超基线15%以上,保留原模型能力
- 适合需精准更新视觉语言模型的科研与工业场景
模型编辑旨在无需耗时的完整重训练即可高效更新预训练模型的知识。现有开创性编辑方法虽成果显著,但主要聚焦于单模态语言模型(LLMs)。对于涉及多模态的视觉语言模型(VLMs),各模态在编辑中的作用与影响仍不明确。我们研究发现:(1)文本与视觉表示在不同层达到敏感峰值,反映其重要性差异;(2)同时编辑双模态可高效更新知识,但会损害模型原有能力。基于此,我们提出DualEdit,分别在关键层修改文本与视觉模态,并引入门控模块增强文本模态敏感性,实现新知识高效注入的同时保护原有信息。我们在多个VLM骨干网络与基准数据集上评估,结果表明DualEdit在多项指标上优于当前最先进的VLM编辑基线及适配的LLM编辑方法。代码已开源。
原文摘要 · Abstract (English)
Model editing aims to efficiently update a pre-trained model's knowledge without the need for time-consuming full retraining. While existing pioneering editing methods achieve promising results, they primarily focus on editing single-modal language models (LLMs). However, for vision-language models (VLMs), which involve multiple modalities, the role and impact of each modality on editing performance remain largely unexplored. To address this gap, we explore the impact of textual and visual modalities on model editing and find that: (1) textual and visual representations reach peak sensitivity at different layers, reflecting their varying importance; and (2) editing both modalities can efficiently update knowledge, but this comes at the cost of compromising the model's original capabilities. Based on our findings, we propose DualEdit, an editor that modifies both textual and visual modalities at their respective key layers. Additionally, we introduce a gating module within the more sensitive textual modality, allowing DualEdit to efficiently update new knowledge while preserving the model's original information. We evaluate DualEdit across multiple VLM backbones and benchmark datasets, demonstrating its superiority over state-of-the-art VLM editing baselines as well as adapted LLM editing methods on different evaluation metrics. Codes are available at https://github.com/zhiyiscs/DualEdit
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。