用强化学习让大模型持续精准修正知识，效率提升超97%。
Reinforced Lifelong Editing for Language Models
- 将编辑过程建模为强化学习，以损失为奖励优化参数更新
- 在多轮编辑中实现59.24%的性能提升，仅需2.11%的耗时
- 适合需要长期知识维护的大模型应用，如智能助手
大型语言模型（LLMs）从预训练语料中获取知识，但其存储信息可能随时间变得不准确或过时。模型编辑通过修改参数而非重新训练来解决此问题，现有方法多采用超网络生成参数更新。然而，这些方法在长期编辑中面临挑战，因其与动态变化的LLM参数不兼容。为此，我们观察到基于超网络的长期编辑可映射为强化学习建模，并提出RLEdit——一种基于强化学习的编辑方法。通过将编辑损失视为奖励，在完整知识序列层面优化超网络参数，使模型能精确捕捉LLM变化并生成合适参数更新。我们在多个LLM上进行充分实验，结果表明RLEdit在长期编辑中表现优于现有方法，兼具更高有效性和效率：性能提升59.24%，耗时仅为多数方法的2.11%。代码已开源：https://github.com/zhrli324/RLEdit。
原文摘要 · Abstract (English)
Large language models (LLMs) acquire information from pre-training corpora, but their stored knowledge can become inaccurate or outdated over time. Model editing addresses this challenge by modifying model parameters without retraining, and prevalent approaches leverage hypernetworks to generate these parameter updates. However, they face significant challenges in lifelong editing due to their incompatibility with LLM parameters that dynamically change during the editing process. To address this, we observed that hypernetwork-based lifelong editing aligns with reinforcement learning modeling and proposed RLEdit, an RL-based editing method. By treating editing losses as rewards and optimizing hypernetwork parameters at the full knowledge sequence level, we enable it to precisely capture LLM changes and generate appropriate parameter updates. Our extensive empirical evaluation across several LLMs demonstrates that RLEdit outperforms existing methods in lifelong editing with superior effectiveness and efficiency, achieving a 59.24% improvement while requiring only 2.11% of the time compared to most approaches. Our code is available at: https://github.com/zhrli324/RLEdit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。