arXiv:2602.06585cs.LGcs.AI2026-02

用随机噪声预训练初始化网络,加速高分辨率学习收敛

Target noise: A pre-training based neural network initialization for efficient high resolution learning

  • 以随机噪声为目标进行自监督预训练,生成结构化权重
  • 在图像修复等任务中,收敛速度提升显著,且无需额外数据
  • 特别适合隐式神经表示,能更早捕捉高频细节

权重初始化对神经网络的优化行为和收敛效率至关重要。现有方法如Xavier、Kaiming初始化依赖随机采样,未能利用优化过程自身的信息。本文提出一种基于随机噪声作为目标的自监督预训练初始化策略:先让网络拟合随机噪声,获得结构化参数配置,再用于下游任务。该方法不需额外数据或修改网络结构,显著加快收敛速度。尤其适用于具有强低频偏好的隐式神经表示(INRs)和Deep Image Prior(DIP)类网络,在噪声预训练后,网络能更早捕获高频成分,实现更快更稳定的优化。尽管随机噪声无语义信息,但其白谱特性使其成为有效的自监督信号。本工作证明,基于噪声的预训练是一种轻量、通用的初始化替代方案,可提升深层神经网络的优化效率。

原文摘要 · Abstract (English)

Weight initialization plays a crucial role in the optimization behavior and convergence efficiency of neural networks. Most existing initialization methods, such as Xavier and Kaiming initializations, rely on random sampling and do not exploit information from the optimization process itself. We propose a simple, yet effective, initialization strategy based on self-supervised pre-training using random noise as the target. Instead of directly training the network from random weights, we first pre-train it to fit random noise, which leads to a structured and non-random parameter configuration. We show that this noise-driven pre-training significantly improves convergence speed in subsequent tasks, without requiring additional data or changes to the network architecture. The proposed method is particularly effective for implicit neural representations (INRs) and Deep Image Prior (DIP)-style networks, which are known to exhibit a strong low-frequency bias during optimization. After noise-based pre-training, the network is able to capture high-frequency components much earlier in training, leading to faster and more stable convergence. Although random noise contains no semantic information, it serves as an effective self-supervised signal (considering its white spectrum nature) for shaping the initialization of neural networks. Overall, this work demonstrates that noise-based pre-training offers a lightweight and general alternative to traditional random initialization, enabling more efficient optimization of deep neural networks.

初始化预训练隐式表征高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。