通过权重缩放加速正弦神经场训练，速度提升10倍
Fast Training of Sinusoidal Neural Fields via Scaling Initialization
- 对正弦神经场的权重进行统一缩放，显著提升训练速度
- 实验表明该方法在多个数据域上实现10倍加速
- 适合追求高效训练的神经场研究者与应用开发者
神经场作为一种新兴范式，将数据表示为由神经网络参数化的连续函数。尽管优势明显，但其训练成本较高，限制了广泛应用。本文聚焦于一类流行的神经场——正弦神经场（SNFs），研究如何通过初始化策略最大化训练速度。发现现有基于信号传播原理的标准初始化方案并非最优。通过简单地将除最后一层外的所有权重乘以一个常数，可使训练速度提升10倍。这一方法称为权重缩放（weight scaling），在多种数据域上均表现出显著加速效果，甚至优于近期提出的其他架构。通过理论与实证分析，我们揭示该方法能有效缓解谱偏差，并具备良好的优化轨迹条件。
原文摘要 · Abstract (English)
Neural fields are an emerging paradigm that represent data as continuous functions parameterized by neural networks. Despite many advantages, neural fields often have a high training cost, which prevents a broader adoption. In this paper, we focus on a popular family of neural fields, called sinusoidal neural fields (SNFs), and study how it should be initialized to maximize the training speed. We find that the standard initialization scheme for SNFs -- designed based on the signal propagation principle -- is suboptimal. In particular, we show that by simply multiplying each weight (except for the last layer) by a constant, we can accelerate SNF training by 10$\times$. This method, coined $\textit{weight scaling}$, consistently provides a significant speedup over various data domains, allowing the SNFs to train faster than more recently proposed architectures. To understand why the weight scaling works well, we conduct extensive theoretical and empirical analyses which reveal that the weight scaling not only resolves the spectral bias quite effectively but also enjoys a well-conditioned optimization trajectory. The code is available $\href{https://github.com/effl-lab/Fast-Neural-Fields}{here}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。