arXiv:2605.06300cs.LG2026-05被引 1

通过提前激活正则化,让神经网络在训练早期就为数据区域生成更多分段线性区域。

Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks

论文配图:Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks
图 1 · 摘自论文原文
  • 设计可插拔正则化项,引导神经元切换面靠近数据点以提前划分输入空间。
  • 在玩具数据集上实测提升分段线性区域数量,并改善早期准确率。
  • 适用于希望增强模型局部表达能力的场景,尤其适合经典架构改进。

具有连续分段仿射激活函数的深度网络会将输入空间划分为多面体区域,其实际产生的仿射区域数量是衡量模型表达能力的关键指标,直接影响对非线性目标函数的逼近效果。实践中,标准训练在数据邻域内实现的区域细分远低于架构理论支持的上限,而现有区域计数理论主要关注架构本身,难以指导优化如何影响数据附近的实际分区。本文提出一个充分条件:当神经元切换面足够接近数据点时,其与局部邻域相交,从而严格增加局部仿射区域数量,为训练过程中早期种子化数据相关分区提供了理论依据。基于此,我们提出一种即插即用的区域播种正则化器,鼓励训练初期进行分区,后期仍允许任务驱动的精细化调整。实验表明,该正则化器通过精确枚举验证了仿射区域数量的增加,在玩具数据集上提升了整体性能;同时显著提高初始阶段准确率,并在ImageNet-1k上使经典模型达到相当或略有提升的最终精度。

原文摘要 · Abstract (English)

Deep networks with continuous piecewise affine activations induce polyhedral partitions of the input space, making the number of realized affine regions a natural measure of expressive capacity and a key determinant of how well the model can approximate nonlinear target functions. In practice, standard training realizes far fewer region refinements in data-visited neighborhoods than the architecture could in principle support, while existing region-count theory is primarily architectural and offers little guidance on how optimization shapes the realized partition near the data. Our theory provides a sufficient condition under which bringing neuron switching surfaces sufficiently close to data points ensures their intersection with local neighborhoods, which in turn implies a strict increase in the local affine-region count, yielding a principled training-time handle for seeding data-relevant partitions early in optimization. Guided by these results, we propose a plug-and-play region-seeding regularizer that encourages early partitioning while allowing task-driven refinement to dominate later in training. Experiments show that the regularizer increases the number of realized affine regions via exact enumeration and improves overall performance on toy datasets, while also improving early-stage accuracy and achieving comparable (or slightly improved) final accuracy on ImageNet-1k for classical models.

神经网络分段线性正则化表达能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。