深度线性网络中，SGD训练使不同模型学出完全旋转等价的表示。
Proof of a perfect platonic representation hypothesis
- 通过分析梯度下降过程中的熵力，证明不同结构网络会收敛到完全柏拉图表示。
- 多数最优解非柏拉图，但SGD总能找到完美柏拉图解，极为罕见。
- 揭示了表示学习与渐进锐化现象共享同一生成机制。
本文详述并解释了Ziyin等人（2025）对嵌入式深度线性网络模型（EDLN）中“完美柏拉图表示假设”（PRH）的证明。我们表明，若使用随机梯度下降（SGD）训练，两个宽度和深度不同且在不同数据上训练的EDLN将变为完全柏拉图，即任意层对均学习到至多旋转等价的表示。由于损失函数的大多数全局最小值并非柏拉图，而SGD仅能发现完美的柏拉图解，这一结果极为不凡。证明还暗示至少六种破坏PRH的方式。此外，在EDLN模型中，柏拉图表示的出现原因与渐进锐化现象相同，表明这两个看似无关的深度学习现象实则具有共同成因。整体而言，该理论与证明强调了理解由SGD训练不可逆性引发的涌现“熵力”及其在表示学习中的作用的重要性。本文旨在深入浅出，避免术语堆砌与冗长技术细节。
原文摘要 · Abstract (English)
In this note, we elaborate on and explain in detail the proof given by Ziyin et al. (2025) of the ``perfect" Platonic Representation Hypothesis (PRH) for the embedded deep linear network model (EDLN). We show that if trained with the stochastic gradient descent (SGD), two EDLNs with different widths and depths and trained on different data will become Perfectly Platonic, meaning that every possible pair of layers will learn the same representation up to a rotation. Because most of the global minima of the loss function are not Platonic, that SGD only finds the perfectly Platonic solution is rather extraordinary. The proof also suggests at least six ways the PRH can be broken. We also show that in the EDLN model, the emergence of the Platonic representations is due to the same reason as the emergence of progressive sharpening. This implies that these two seemingly unrelated phenomena in deep learning can, surprisingly, have a common cause. Overall, the theory and proof highlight the importance of understanding emergent "entropic forces" due to the irreversibility of SGD training and their role in representation learning. The goal of this note is to be instructive while avoiding jargon and lengthy technical details.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。