arXiv:2508.09330cs.LGcs.AI2025-08被引 1

受生物突触修剪启发,动态剪枝提升模型效率与预测精度。

Synaptic Pruning: A Biological Inspiration for Deep Learning Regularization

  • 按权重绝对值大小逐步剪除低重要性连接,模拟生物神经机制。
  • 在金融时间序列预测中误差降低最高达52%,显著优于传统dropout。
  • 无需额外微调阶段,可直接嵌入训练流程,适配多种模型架构。

生物大脑中的突触修剪通过删除弱连接来提升效率。相比之下,人工神经网络中的随机丢弃(dropout)不考虑活动依赖性。本文提出一种基于权重幅度的渐进式突触剪枝方法,更贴近生物学原理:在训练过程中逐步移除低重要性连接。该方法直接集成于训练循环中,替代传统dropout,通过各层权重绝对值计算重要性,并采用立方函数调度逐步提高全局稀疏度。在固定间隔,剪枝掩码永久移除低重要性权重,同时保持活跃权重的梯度流,避免了单独剪枝与微调阶段。在RNN、LSTM及Patch Time Series Transformer等多种时序模型上,针对四个数据集的实验均显示一致性能提升。整体表现最佳,经弗里德曼检验确认差异显著(p < 0.01)。在金融预测任务中,相较无剪枝或标准dropout模型,平均绝对误差最高降低20%;在部分Transformer模型中降幅高达52%。该动态剪枝机制将权重消减与渐进稀疏化结合,为正则化提供新范式,尤其在金融时间序列预测中表现出强大潜力,是传统dropout的有效实用替代方案。

原文摘要 · Abstract (English)

Synaptic pruning in biological brains removes weak connections to improve efficiency. In contrast, dropout regularization in artificial neural networks randomly deactivates neurons without considering activity-dependent pruning. We propose a magnitude-based synaptic pruning method that better reflects biology by progressively removing low-importance connections during training. Integrated directly into the training loop as a dropout replacement, our approach computes weight importance from absolute magnitudes across layers and applies a cubic schedule to gradually increase global sparsity. At fixed intervals, pruning masks permanently remove low-importance weights while maintaining gradient flow for active ones, eliminating the need for separate pruning and fine-tuning phases. Experiments on multiple time series forecasting models including RNN, LSTM, and Patch Time Series Transformer across four datasets show consistent gains. Our method ranked best overall, with statistically significant improvements confirmed by Friedman tests (p < 0.01). In financial forecasting, it reduced Mean Absolute Error by up to 20% over models with no or standard dropout, and up to 52% in select transformer models. This dynamic pruning mechanism advances regularization by coupling weight elimination with progressive sparsification, offering easy integration into diverse architectures. Its strong performance, especially in financial time series forecasting, highlights its potential as a practical alternative to conventional dropout techniques.

深度学习正则化剪枝时间序列

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。