统一多模态知识编辑，让模型既准又稳还不会乱改。
Towards Unified Multimodal Editing with Enhanced Knowledge Collaboration
- 把内部和外部知识都当记忆库，用认知类比方式更新。
- 编辑后模型在可靠性、泛化性和局部性上全达标。
- 适合需要精准可控知识更新的多模态应用。
多模态大模型（MLLMs）的快速发展带来了有效的知识编辑挑战。现有方法包括内在知识编辑和外部知识调用，各有优劣，难以同时兼顾可靠性、泛化性和局部性。本文提出UniKE，一种统一的多模态编辑方法，将两类知识视为向量化的键值记忆，其编辑过程类比人类认知中的同化与顺应阶段，均在相同语义层级进行。在此统一框架下，通过将知识表示解耦至语义空间与真实性空间，进一步促进知识协同。大量实验验证了该方法的有效性，使编辑后的MLLM在可靠性、泛化性和局部性方面均表现优异。UniKE代码已开源：https://github.com/beepkh/UniKE。
原文摘要 · Abstract (English)
The swift advancement in Multimodal LLMs (MLLMs) also presents significant challenges for effective knowledge editing. Current methods, including intrinsic knowledge editing and external knowledge resorting, each possess strengths and weaknesses, struggling to balance the desired properties of reliability, generality, and locality when applied to MLLMs. In this paper, we propose UniKE, a novel multimodal editing method that establishes a unified perspective and paradigm for intrinsic knowledge editing and external knowledge resorting. Both types of knowledge are conceptualized as vectorized key-value memories, with the corresponding editing processes resembling the assimilation and accommodation phases of human cognition, conducted at the same semantic levels. Within such a unified framework, we further promote knowledge collaboration by disentangling the knowledge representations into the semantic and truthfulness spaces. Extensive experiments validate the effectiveness of our method, which ensures that the post-edit MLLM simultaneously maintains excellent reliability, generality, and locality. The code for UniKE is available at \url{https://github.com/beepkh/UniKE}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。