L2正则让神经网络权重呈现低秩结构，可实现高效模型合并。
Low-rank bias, weight decay, and model merging in neural networks
- 通过L2正则分析权重矩阵的低秩特性，揭示参数与梯度对齐规律。
- 实验验证低秩偏差在多种网络中稳定存在，且不依赖严格假设。
- 输入正交时，直接加权可融合双任务，性能接近独立训练模型。
我们研究了在带L2正则（即权重衰减）的优化算法极限解下，神经网络权重矩阵的低秩结构。结果显示，在驻点处参数与梯度对齐、层间范数保持不变，以及低秩偏差等性质，这些现象此前仅在梯度下降/流算法解中被观察到。实验表明，分析中的假设对观测结果影响甚微。此外，我们探究了由L2正则和低秩偏差引发的多任务学习现象：当两个网络的训练输入近似正交时，直接将它们的权重相加即可得到新网络，该网络在两个训练集上的表现均接近各自独立训练的性能。此现象在浅层ReLU网络（梯度下降训练）及深层线性网络（梯度流训练）中均得到验证。
原文摘要 · Abstract (English)
We explore the low-rank structure of the weight matrices in neural networks at the stationary points (limiting solutions of optimization algorithms) with $L2$ regularization (also known as weight decay). We show several properties of such deep neural networks, induced by $L2$ regularization. In particular, for a stationary point we show alignment of the parameters and the gradient, norm preservation across layers, and low-rank bias: properties previously known in the context of solution of gradient descent/flow type algorithms. Experiments show that the assumptions made in the analysis only mildly affect the observations. In addition, we investigate a multitask learning phenomenon enabled by $L2$ regularization and low-rank bias. In particular, we show that if two networks are trained, such that the inputs in the training set of one network are approximately orthogonal to the inputs in the training set of the other network, the new network obtained by simply summing the weights of the two networks will perform as well on both training sets as the respective individual networks. We demonstrate this for shallow ReLU neural networks trained by gradient descent, as well as deep linear networks trained by gradient flow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。