arXiv:2411.15432cs.CLcs.CV2024-11CVPR被引 18

提出LiveEdit,实现视觉语言模型的持续知识更新。

Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts

  • 用低秩专家生成器为每次编辑独立生成专家模块。
  • 通过视觉语义过滤和文本语义路由,实现多专家融合。
  • 首个专用于视觉语言模型的持续编辑框架,适合需要长期维护的AI系统。

模型编辑旨在不重新训练的前提下修正错误知识、更新过时信息并融入新数据。这一任务在需持续更新的现实场景中面临挑战。尽管现有编辑方法在纯语言模型中表现稳健,但包含视觉模态的视觉语言模型(VLLMs)难以直接适配。本文提出LiveEdit,一种面向视觉语言模型的持续编辑框架。首先训练编辑专家生成器,为每个编辑实例独立生成低秩专家,以修正VLLM的响应。引入硬过滤机制,利用视觉语义知识在推理阶段粗略剔除与输入无关的专家。最后,基于文本语义相关性设计软路由机制,实现视觉相关专家的融合。我们建立首个终身视觉语言模型编辑基准。大量实验表明,LiveEdit在终身编辑场景中显著优于基线。模块消融实验验证了各设计的有效性。

原文摘要 · Abstract (English)

Model editing aims to correct inaccurate knowledge, update outdated information, and incorporate new data into Large Language Models (LLMs) without the need for retraining. This task poses challenges in lifelong scenarios where edits must be continuously applied for real-world applications. While some editors demonstrate strong robustness for lifelong editing in pure LLMs, Vision LLMs (VLLMs), which incorporate an additional vision modality, are not directly adaptable to existing LLM editors. In this paper, we propose LiveEdit, a LIfelong Vision language modEl Edit to bridge the gap between lifelong LLM editing and VLLMs. We begin by training an editing expert generator to independently produce low-rank experts for each editing instance, with the goal of correcting the relevant responses of the VLLM. A hard filtering mechanism is developed to utilize visual semantic knowledge, thereby coarsely eliminating visually irrelevant experts for input queries during the inference stage of the post-edited model. Finally, to integrate visually relevant experts, we introduce a soft routing mechanism based on textual semantic relevance to achieve multi-expert fusion. For evaluation, we establish a benchmark for lifelong VLLM editing. Extensive experiments demonstrate that LiveEdit offers significant advantages in lifelong VLLM editing scenarios. Further experiments validate the rationality and effectiveness of each module design in LiveEdit.

视觉语言模型知识编辑持续学习专家混合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。