提出新型梯度下降方法,突破传统正则化限制,实现更优的模型泛化性能。
Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent

- 通过早期停止的负移梯度下降,实现混合符号谱正则化。
- 在高有效秩尾部下,风险降低多项式因子,优于所有可接受终点。
- 适合处理过参数化线性回归中的复杂谱结构,对算法设计有启发意义。
在过参数化线性回归中,许多弱谱方向类似于对信号主导谱的岭正则化;负岭是自然修正,使滤波器值超过1。然而,稳定的负岭终点存在结构性限制:其极点必须低于最小非零经验特征值,且对较小特征值的反收缩强于较大者。早期停止的负移梯度下降可突破此约束:其滤波器在潜在极点处平滑,具备混合符号能力——无岭方向构成主导前缀,低频方向被压缩或暴露可控,而停止时机决定交叉点。在高斯尖峰加平坦模型中,我们发现马尔琴科-帕斯图尔障碍:抵消隐式惩罚所需的偏移位于最小经验特征值之上一个主干宽度处,停止路径在显式条件下,风险改进幅度达到多项式因子。主定理允许一般高有效秩尾部:其迹决定隐式下界,平方谱控制暴露,下界临界路径可一次性恢复所有头部尺度,超越正收缩,一旦尺度分离,亦可实现所有均匀重缩放的无岭情形。处理非收缩的移位动力学是核心技术挑战;局部杜哈梅积分用于控制之。有限网格留出不等式将分离性传递至验证选择的算法。
原文摘要 · Abstract (English)
In overparameterized linear regression, many weak spectral directions act like a ridge penalty on the signal-bearing spectrum; negative ridge is the natural correction, pushing filters above one. The stable negative-ridge endpoint, however, is structurally limited: its pole must stay below the smallest nonzero empirical eigenvalue, and it anti-shrinks smaller eigenvalues more than larger ones. Early-stopped negative-shifted gradient descent escapes this constraint. Its filter is smooth at the would-be pole and mixed-sign-capable: above-ridgeless directions form a leading prefix, with lower directions shrunk or exposure-controlled while stopping sets the crossover. In a Gaussian spike-plus-flat model we discover a Marchenko-Pastur barrier: the shift that cancels the implicit penalty lies a bulk width above the smallest empirical eigenvalue, and the stopped path improves on every admissible endpoint by a polynomial factor in risk under explicit conditions. Our main theorem permits a general high-effective-rank tail: its trace sets the implicit floor, its squared spectrum controls exposure, and the floor-critical path recovers all head scales at once, beyond positive shrinkage and, once scales separate, every uniform rescaling of ridgeless. Handling the noncontractive shifted dynamics is the central technical challenge; localized Duhamel integrals control them. A finite-grid hold-out inequality transfers the separations to the validation-selected algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。