arXiv:2605.15463cs.LG2026-05被引 1

用可微分正则化提升神经网络稳定性,兼顾精度与效率。

Layer-wise Derivative Controlled Networks

论文配图:Layer-wise Derivative Controlled Networks
图 1 · 摘自论文原文
  • 以微分正则化控制中间层导数,替代传统激活函数。
  • MNIST上梯度波动降低23.1%,参数量减少15.5倍。
  • 适合对输出稳定性和硬件效率要求高的部署场景。

随着机器学习模型复杂度提升,高精度、硬件效率和功能稳定性三者难以兼顾。传统架构常因输入微小变化导致输出剧烈波动,限制了在敏感场景的应用。本文提出ChainzRule(CR)架构,将标准分段线性激活替换为由微分正则化(DREG)驱动的多项式引擎。DREG对中间层导数实施精准约束,抑制极端敏感性,同时保留多项式引擎的表征能力。在头对头的'公平对决'基准测试中,ChainzRule在仅使用15.5倍更少参数的情况下超越标准模型。在MNIST数据集上,峰值梯度波动平均降低23.1%,确保更平滑、可预测的流形结构。在Yelp Full有序回归任务中,采用显式DREG正则化的模型达到70.17%准确率,验证了导数感知正则化在真实任务中的有效性。通过将梯度感知嵌入架构,ChainzRule证明稳定性与精度并非对立目标。

原文摘要 · Abstract (English)

As machine learning models grow in complexity, they increasingly struggle with three conflicting demands: the need for high accuracy, the requirement for hardware efficiency, and the necessity of functional stability. Traditional architectures often achieve performance at the expense of spiky or unpredictable behavior, where small changes in input lead to massive swings in output -- a critical flaw for real-world deployment in sensitive environments. This paper introduces ChainzRule (CR), a novel neural architecture designed to harmonize these competing goals. ChainzRule replaces standard piecewise-linear activations with a Polynomial Engine governed by Differential Regularization (DREG). Unlike traditional methods that impose global, coarse-grained constraints on a model's Lipschitz constant, DREG acts as a targeted regularization on intermediate derivatives. This approach suppresses extreme sensitivity without attenuating the representational power inherent in the Polynomial Engine. In head-to-head "Fair Fight" benchmarks, ChainzRule outperformed standard models while using 15.5x fewer parameters. On the MNIST dataset, it reduced peak gradient volatility by an average of 23.1%, ensuring a smoother and more predictable manifold. On Yelp Full ordinal regression under explicit DREG regularization, ChainzRule achieves 70.17% accuracy, validating that derivative-aware regularization is compatible with competitive performance on realistic tasks. By embedding gradient awareness into the architecture via DREG, ChainzRule demonstrates that stability and accuracy need not be competing objectives.

神经网络稳定性正则化高效模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。