通过梯度掩码让神经网络局部化能力，提升可解释性与可控性。
Gradient Routing: Masking Gradients to Localize Computation in Neural Networks
- 训练时用数据依赖的掩码控制梯度流向，指定哪些参数由哪些数据更新。
- 实现可解释的特征表示、鲁棒的遗忘机制和行为模块的可监督性。
- 适用于数据稀缺场景，尤其适合对安全性要求高的实际应用。
神经网络主要基于输入输出进行训练，忽视了内部机制，而这些机制直接影响安全属性，如透明性、无敏感信息或有害能力、以及超出训练分布的可靠泛化。为此，我们提出梯度路由，一种将能力限制在神经网络特定子区域的训练方法。该方法在反向传播中对梯度施加由用户提供的、数据相关的加权掩码，以配置哪些参数由哪些数据点更新。实验表明，梯度路由可实现：(1) 可解释地划分表示；(2) 通过擦除预设网络子区域实现稳健的遗忘；(3) 通过定位负责不同行为的模块，实现强化学习者的可扩展监督。即使仅应用于有限的、临时的数据子集，梯度路由仍能有效定位能力。结果表明，该方法在高质量数据稀缺的真实场景中具有应用前景。
原文摘要 · Abstract (English)
Neural networks are trained primarily based on their inputs and outputs, without regard for their internal mechanisms. These neglected mechanisms determine properties that are critical for safety, like (i) transparency; (ii) the absence of sensitive information or harmful capabilities; and (iii) reliable generalization of goals beyond the training distribution. To address this shortcoming, we introduce gradient routing, a training method that isolates capabilities to specific subregions of a neural network. Gradient routing applies data-dependent, weighted masks to gradients during backpropagation. These masks are supplied by the user in order to configure which parameters are updated by which data points. We show that gradient routing can be used to (1) learn representations which are partitioned in an interpretable way; (2) enable robust unlearning via ablation of a pre-specified network subregion; and (3) achieve scalable oversight of a reinforcement learner by localizing modules responsible for different behaviors. Throughout, we find that gradient routing localizes capabilities even when applied to a limited, ad-hoc subset of the data. We conclude that the approach holds promise for challenging, real-world applications where quality data are scarce.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。