打乱96层神经网络后，仍能准确复原其顺序。
I Dropped a Neural Net
- 通过输入输出投影的对角优势比判断层间配对关系。
- 在约10^122种组合中，仅用少量迭代即恢复正确顺序。
- 适合对模型可解释性与训练稳定性感兴趣的读者。
一篇近期由Dwarkesh Patel采访中提到的Jane Street难题：他们训练了一个残差网络，随机打乱了全部96层，并要求重新排序。给定未标记的残差网络层及其训练数据，我们能够精确复原层的原始顺序。问题分解为两部分：每对块的输入/输出投影配对（48!种可能）和重新组装后的块序排列（48!种可能），联合搜索空间约为(48!)² ≈ 10¹²²，远超可观测宇宙中的原子数。我们发现训练过程中的稳定性条件（如动态等变性）会使正确配对层的输出-输入权值乘积 $W_{ ext{out}} W_{ ext{in}}$ 呈负对角结构，从而利用对角占优比作为配对信号。对于排序，我们以delta-norm或$ orm{W_{ ext{out}}}_F$为初始代理，通过梯度上升逼近零均方误差。
原文摘要 · Abstract (English)
A recent Dwarkesh Patel podcast with John Collison and Elon Musk featured an interesting puzzle from Jane Street: they trained a neural net, shuffled all 96 layers, and asked to put them back in order. Given unlabelled layers of a Residual Network and its training dataset, we recover the exact ordering of the layers. The problem decomposes into pairing each block's input and output projections ($48!$ possibilities) and ordering the reassembled blocks ($48!$ possibilities), for a combined search space of $(48!)^2 \approx 10^{122}$, which is more than the atoms in the observable universe. We show that stability conditions during training like dynamic isometry leave the product $W_{\text{out}} W_{\text{in}}$ for correctly paired layers with a negative diagonal structure, allowing us to use diagonal dominance ratio as a signal for pairing. For ordering, we seed-initialize with a rough proxy such as delta-norm or $\|W_{\text{out}}\|_F$ then hill-climb to zero mean squared error.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。