arXiv:2606.13461cs.LGcs.CV2026-06

用强化学习自动编辑神经网络,省去手动设计算法的麻烦。

Reinforcement Learning for Neural Model Editing

论文配图:Reinforcement Learning for Neural Model Editing
图 1 · 摘自论文原文
  • 将模型编辑转化为强化学习问题,通过奖励反馈指导修改。
  • 在图像去记忆任务中,遗忘集准确率接近0%,保留集超90%。
  • 适合需要自动化编辑且兼顾性能的场景,如偏见消除与数据删除。

神经网络编辑需针对特定目标设计专用算法,但过程耗时且费力。本文提出一种探索性框架,将模型编辑建模为强化学习问题,智能体通过奖励反馈修改模型。引入两种环境:MaskWorld(乘法缩放权重)和ShiftWorld(加法更新权重)。奖励函数结合效用保持目标与任务特定编辑目标,使智能体在维持整体性能的同时学习精准修改策略。我们在文本分类的偏见缓解与图像分类的机器去记忆任务上进行评估,二者传统上依赖专用算法。结果表明,所学策略在去记忆任务中将遗忘集准确率降至近0%,同时保留集准确率超过90%;在偏见缓解任务中,偏见相关性能提升超5%,且通用分类性能保持稳定。研究证明,神经网络编辑可转化为强化学习问题,使编辑策略从奖励反馈中学习,无需为每项任务手工设计。

原文摘要 · Abstract (English)

Editing pretrained neural networks requires specialized algorithms tailored to specific objectives. Designing such algorithms is often time-consuming and demands significant effort. We present an exploratory framework that formulates neural model editing as a reinforcement learning problem, where agents modify models using reward feedback. We introduce two environments: MaskWorld, where agents scale weights multiplicatively, and ShiftWorld, where agents apply additive weight updates. The reward function combines a utility-preservation objective with a task-specific editing objective, enabling agents to learn targeted modifications while maintaining overall model performance. We evaluate the framework on bias mitigation in text classification and machine unlearning in image classification, both of which traditionally rely on specialized algorithms. Our results show that the learned policies reduce forget set accuracy to nearly 0% while preserving over 90% retain set accuracy on the unlearning task. In the bias mitigation setting, the learned policies improve bias-related performance by more than 5% while maintaining general classification utility. Our findings show that neural model editing can be cast as a reinforcement learning problem, allowing editing policies to be learned from reward feedback rather than manually engineered for each task.

强化学习模型编辑去记忆偏见缓解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。