SIREN网络存在频谱瓶颈,噪声初始化可显著提升拟合精度。
Spectral Bottleneck in Sinusoidal Representation Networks: Noise is All You Need
- 通过分析激活频谱与神经正切核,发现初始化不当导致频域能量分布失衡
- 加入高斯扰动后,频谱与神经正切核特性改善,输出零值问题缓解
- 提出目标感知初始化方法WINNER,适用于音频与图像高精度重建
本文揭示了使用正弦激活函数的隐式神经表示中一个根本性局限:SIREN的拟合误差对目标频率内容和初始化方式极为敏感。极端情况下,这种敏感性会导致频谱瓶颈,使输出变为零。通过分析训练过程中激活频谱和经验神经正切核(NTK)的演变,发现频域能量分布不均是该失效模式的根源。研究进一步考察了在基础均匀初始化权重上施加高斯扰动的影响,表明这些扰动能有效调节激活频谱和NTK特征基。整体表明,初始化是决定SIREN演化路径的核心因素,随着目标长度增加与精细细节需求提高,亟需采用自适应、目标感知的策略。所提出的权重初始化方案(WINNER)是一种简单但有效的改进,通过调整网络激活的频谱特性显著提升拟合精度,在音频拟合任务上达到当前最优性能,并在图像拟合任务中取得显著进步。
原文摘要 · Abstract (English)
This work identifies and attempts to address a fundamental limitation of implicit neural representations with sinusoidal activation. The fitting error of SIRENs is highly sensitive to the target frequency content and to the choice of initialization. In extreme cases, this sensitivity leads to a spectral bottleneck that can result in a zero-valued output. This phenomenon is characterized by analyzing the evolution of activation spectra and the empirical neural tangent kernel (NTK) during the training process. An unfavorable distribution of energy across frequency modes was noted to give rise to this failure mode. Furthermore, the effect of Gaussian perturbations applied to the baseline uniformly initialized weights is examined, showing how these perturbations influence activation spectra and the NTK eigenbasis of SIREN. Overall, initialization emerges as a central factor governing the evolution of SIRENs, indicating the need for adaptive, target-aware strategies as the target length increases and fine-scale detail becomes essential. The proposed weight initialization scheme (WINNER) represents a simple ad hoc step in this direction and demonstrates that fitting accuracy can be significantly improved by modifying the spectral profile of network activations through a target-aware initialization. The approach achieves state-of-the-art performance on audio fitting tasks and yields notable improvements in image fitting tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。