arXiv:2505.11589cs.LG2025-05被引 1

提出新训练框架,让多项式神经网络稳定高效训练。

A Training Framework for Optimal and Stable Training of Polynomial Neural Networks

  • 用边界损失抑制激活值溢出,提升数值稳定性。
  • 低阶多项式(如2次)即达高精度,最高支持22次多项式。
  • 适合需要加密推理的隐私保护场景,兼容同态加密。

通过用多项式激活替代标准非线性函数,多项式神经网络(PNNs)在同态加密(HE)支持的隐私保护推理中具有重要意义。然而,有效训练PNN面临挑战:低阶多项式限制模型表达能力,而高阶多项式虽能捕捉复杂函数,却常导致数值不稳定性与梯度爆炸。本文提出一种鲁棒且通用的训练框架,包含两项协同创新:1)新型边界损失,对超出预设稳定范围的激活输入进行指数级惩罚;2)选择性梯度裁剪,在保持批归一化统计量的同时有效控制梯度大小。我们在多种图像、音频及人体活动识别数据集上,使用由HE兼容层(如线性层、平均池化、批归一化等)构成的深层架构验证了该框架的广泛有效性。所训练的PNN在低阶多项式(如2次)下即实现高准确率,关键是在多项式阶数高达22时仍保持稳定训练与优异性能,而传统方法在此情况下通常失效或严重退化。此外,这些PNN的性能与原始ReLU基模型高度接近,达到显著的近似效果。大量消融实验验证了各项技术的贡献,并指导超参数选择。我们确认训练模型具备HE兼容性,推动了高精度、稳定且安全的深度学习推理的实际部署。

原文摘要 · Abstract (English)

By replacing standard non-linearities with polynomial activations, Polynomial Neural Networks (PNNs) are pivotal for applications such as privacy-preserving inference via Homomorphic Encryption (HE). However, training PNNs effectively presents a significant challenge: low-degree polynomials can limit model expressivity, while higher-degree polynomials, crucial for capturing complex functions, often suffer from numerical instability and gradient explosion. We introduce a robust and versatile training framework featuring two synergistic innovations: 1) a novel Boundary Loss that exponentially penalizes activation inputs outside a predefined stable range, and 2) Selective Gradient Clipping that effectively tames gradient magnitudes while preserving essential Batch Normalization statistics. We demonstrate our framework's broad efficacy by training PNNs within deep architectures composed of HE-compatible layers (e.g., linear layers, average pooling, batch normalization, as used in ResNet variants) across diverse image, audio, and human activity recognition datasets. These models consistently achieve high accuracy with low-degree polynomial activations (such as degree 2) and, critically, exhibit stable training and strong performance with polynomial degrees up to 22, where standard methods typically fail or suffer severe degradation. Furthermore, the performance of these PNNs achieves a remarkable parity, closely approaching that of their original ReLU-based counterparts. Extensive ablation studies validate the contributions of our techniques and guide hyperparameter selection. We confirm the HE-compatibility of the trained models, advancing the practical deployment of accurate, stable, and secure deep learning inference.

多项式网络加密推理训练稳定同态加密

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。