arXiv:2602.01526cs.LG2026-02

揭示隐式神经表示中输入秩坍缩问题,提出无需修改结构的初始化修复方法。

The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedy

  • 通过层间分解发现首层输入秩不足导致表达瓶颈。
  • 新初始化方法使模型在不增加计算量下实现高保真重建。
  • 为位置编码、SIREN等方法提供统一理论解释,适合研究者参考。

隐式神经表示(INRs)在连续信号建模中取得突破,但在有限训练预算下难以恢复细节。尽管位置编码(PE)、正弦激活(SIREN)和批归一化(BN)等经验性方法有效缓解此问题,其理论解释多为事后分析,仅关注修改后的全局神经切线核(NTK)谱。本文反其道而行之,提出结构诊断框架。通过层间分解NTK,数学上识别出“入口秩坍缩”:低维输入坐标无法覆盖高维嵌入空间,导致首层存在根本性秩缺陷,成为全网表达瓶颈。该框架将PE、SIREN与BN统一为不同形式的秩恢复机制。基于此诊断,我们推导出一种秩扩展初始化策略,无需架构改动或额外计算开销,即可保证表示秩随层宽增长。实验表明,该原则性修复使标准MLP实现高保真重建,证明提升INRs性能的关键在于初始秩传播的结构性优化,以有效填充潜在空间。

原文摘要 · Abstract (English)

Implicit Neural Representations (INRs) have revolutionized continuous signal modeling, yet they struggle to recover fine-grained details within finite training budgets. While empirical techniques, such as positional encoding (PE), sinusoidal activations (SIREN), and batch normalization (BN), effectively mitigate this, their theoretical justifications are predominantly post hoc, focusing on the global NTK spectrum only after modifications are applied. In this work, we reverse this paradigm by introducing a structural diagnostic framework. By performing a layer-wise decomposition of the NTK, we mathematically identify the ``Inlet Rank Collapse'': a phenomenon where the low-dimensional input coordinates fail to span the high-dimensional embedding space, creating a fundamental rank deficiency at the first layer that acts as an expressive bottleneck for the entire network. This framework provides a unified perspective to re-interpret PE, SIREN, and BN as different forms of rank restoration. Guided by this diagnosis, we derive a Rank-Expanding Initialization, a minimalist remedy that ensures the representation rank scales with the layer width without architectural modifications or computational overhead. Our results demonstrate that this principled remedy enables standard MLPs to achieve high-fidelity reconstructions, proving that the key to empowering INRs lies in the structural optimization of the initial rank propagation to effectively populate the latent space.

隐式神经表示秩坍缩初始化理论分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。