通过凸松弛方法，为多分支齐次神经网络提供近似线性的样本复杂度泛化界。
A Convex Relaxation Approach to Generalization Analysis for Parallel Positively Homogeneous Networks
- 将非凸经验风险最小化转化为相关凸优化问题，获得全局下界。
- 在多种模型上实现样本复杂度几乎随网络宽度线性增长的泛化界。
- 适用于矩阵感知、注意力机制等广泛结构，适合理论研究者参考。
我们提出一个通用框架，用于推导并行正齐次神经网络的泛化界——这类网络的输入输出映射可分解为若干正齐次映射之和。典型例子包括矩阵分解与感知、单层多头注意力机制、张量分解、深度线性与ReLU网络等。该框架通过将非凸的经验风险最小化（ERM）问题与一个相关的凸优化问题关联，获得对原始非凸问题的全局可实现下界。利用这一凸下界，在凸空间中进行泛化分析，并控制凸模型与非凸模型之间的差异。我们将该框架应用于多种模型，涵盖低秩矩阵感知、结构化矩阵感知、两层线性网络、两层ReLU网络及单层多头注意力机制，均获得了样本复杂度几乎随网络宽度线性增长的泛化界。
原文摘要 · Abstract (English)
We propose a general framework for deriving generalization bounds for parallel positively homogeneous neural networks--a class of neural networks whose input-output map decomposes as the sum of positively homogeneous maps. Examples of such networks include matrix factorization and sensing, single-layer multi-head attention mechanisms, tensor factorization, deep linear and ReLU networks, and more. Our general framework is based on linking the non-convex empirical risk minimization (ERM) problem to a closely related convex optimization problem over prediction functions, which provides a global, achievable lower-bound to the ERM problem. We exploit this convex lower-bound to perform generalization analysis in the convex space while controlling the discrepancy between the convex model and its non-convex counterpart. We apply our general framework to a wide variety of models ranging from low-rank matrix sensing, to structured matrix sensing, two-layer linear networks, two-layer ReLU networks, and single-layer multi-head attention mechanisms, achieving generalization bounds with a sample complexity that scales almost linearly with the network width.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。