多任务学习让ReLU神经网络解唯一，且等价于核方法。
The Effects of Multi-Task Learning on ReLU Neural Network Functions
- 多任务时解唯一，对应Sobolev空间最小范数插值。
- 单任务解不唯一，多任务使解趋于唯一。
- 适合研究神经网络与核方法关系的学者。
本文研究浅层ReLU神经网络在多任务学习中的解的性质，网络通过最小化平方权重和来拟合数据。令人惊讶的是，每个任务的解类似于求解核回归问题，揭示了神经网络与核方法的新关联。已知单任务神经网络学习等价于非Hilbert空间中的最小范数插值问题，解通常不唯一；而本文证明，在一维输入的多任务情况下，解几乎总是唯一的，且等同于Sobolev（再生核）希尔伯特空间中的最小范数插值解。我们还展示了高维输入情形下的类似现象：当任务数量大时，神经网络学习问题近似等价于由最优神经元决定的固定核上的ℓ²最小化问题。
原文摘要 · Abstract (English)
This paper studies the properties of solutions to multi-task shallow ReLU neural network learning problems, wherein the network is trained to fit a dataset with minimal sum of squared weights. Remarkably, the solutions learned for each individual task resemble those obtained by solving a kernel regression problem, revealing a novel connection between neural networks and kernel methods. It is known that single-task neural network learning problems are equivalent to a minimum norm interpolation problem in a non-Hilbertian Banach space, and that the solutions of such problems are generally non-unique. In contrast, we prove that the solutions to univariate-input, multi-task neural network interpolation problems are almost always unique, and coincide with the solution to a minimum-norm interpolation problem in a Sobolev (Reproducing Kernel) Hilbert Space. We also demonstrate a similar phenomenon in the multivariate-input case; specifically, we show that neural network learning problems with large numbers of tasks are approximately equivalent to an $\ell^2$ (Hilbert space) minimization problem over a fixed kernel determined by the optimal neurons.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。