用正弦函数结构化初始化，提升模型训练速度与精度。
Sinusoidal Initialization, Time for a New Start
- 采用正弦函数构建权重矩阵，实现确定性初始化
- 平均提升4.9%验证准确率，收敛速度加快20.9%
- 适合追求稳定高效训练的CNN、ViT和大模型应用
初始化在深度神经网络训练中至关重要，直接影响收敛性、稳定性与泛化能力。传统方法如Glorot和He初始化依赖随机性，易导致各层权重分布不均。本文提出正弦初始化（Sinusoidal initialization），一种新型确定性方法，通过正弦函数构造有结构的权重矩阵，有效提升网络中权重分布的均匀性与平衡性，同时使神经元激活状态从首次前向传播起就趋于均匀、良好条件化。由于初始权重与激活已均衡高效利用，该方法在多种模型（包括卷积神经网络、视觉变换器及大语言模型）上均表现出更快速的收敛、更强的训练稳定性与更高的最终精度。实验显示，平均验证准确率提升4.9%，收敛速度提高20.9%。通过以结构替代随机性，该初始化为深度学习系统提供了更稳固可靠的起点。
原文摘要 · Abstract (English)
Initialization plays a critical role in Deep Neural Network training, directly influencing convergence, stability, and generalization. Common approaches such as Glorot and He initializations rely on randomness, which can produce uneven weight distributions across layer connections. In this paper, we introduce the Sinusoidal initialization, a novel deterministic method that employs sinusoidal functions to construct structured weight matrices expressly to improve the spread and balance of weights throughout the network while simultaneously fostering a more uniform, well-conditioned distribution of neuron activation states from the very first forward pass. Because Sinusoidal initialization begins with weights and activations that are already evenly and efficiently utilized, it delivers consistently faster convergence, greater training stability, and higher final accuracy across a wide range of models, including convolutional neural networks, vision transformers, and large language models. On average, our experiments show an increase of 4.9% in final validation accuracy and 20.9% in convergence speed. By replacing randomness with structure, this initialization provides a stronger and more reliable foundation for Deep Learning systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。