让大模型更新地理知识时,不破坏原有能力。
GeoEdit: Geometric Knowledge Editing for Large Language Models
- 利用参数更新的几何关系区分新知识与通用知识对应神经元。
- 在保持通用能力前提下,精准更新目标知识,效果优于现有方法。
- 适合需要持续更新知识的大模型应用,如智能地图、教育助手。
大语言模型的知识更新至关重要,但现有训练类编辑方法常难以在引入新知识的同时保持原有通用知识。为此,我们提出几何知识编辑框架GeoEdit,通过分析微调中参数更新的几何关系,识别与新知识相关的神经元和与通用知识扰动相关的神经元。采用方向感知的知识识别方法,避免更新与已有知识方向近似正交的神经元,从而保护模型泛化能力。对剩余神经元,对齐方向融合新旧知识,对反向方向采用“先遗忘再学习”策略。此外,引入重要性引导的任务向量融合技术,过滤冗余信息并实现自适应神经元加权,进一步提升编辑性能。在两个公开数据集上的大量实验表明,GeoEdit显著优于现有最先进方法。
原文摘要 · Abstract (English)
Regular updates are essential for maintaining up-to-date knowledge in large language models (LLMs). Consequently, various model editing methods have been developed to update specific knowledge within LLMs. However, training-based approaches often struggle to effectively incorporate new knowledge while preserving unrelated general knowledge. To address this challenge, we propose a novel framework called Geometric Knowledge Editing (GeoEdit). GeoEdit utilizes the geometric relationships of parameter updates from fine-tuning to differentiate between neurons associated with new knowledge updates and those related to general knowledge perturbations. By employing a direction-aware knowledge identification method, we avoid updating neurons with directions approximately orthogonal to existing knowledge, thus preserving the model's generalization ability. For the remaining neurons, we integrate both old and new knowledge for aligned directions and apply a "forget-then-learn" editing strategy for opposite directions. Additionally, we introduce an importance-guided task vector fusion technique that filters out redundant information and provides adaptive neuron-level weighting, further enhancing model editing performance. Extensive experiments on two publicly available datasets demonstrate the superiority of GeoEdit over existing state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。