语言模型的内部表示方差实为信息存储，非神经坍缩。
Neural Collapse Is Forbidden: Information Floors in Language Models
- 方差分配遵循信息守恒定律，类别内上下文占79-91%
- 跨14个模型验证，类别结构仅贡献4-12%表示方差
- 适用于理解预训练中信息演化，适合模型分析研究者
语言模型表示中的类内方差常被误读为不完全的神经坍缩。本文指出其本质是信息存储，并遵循特定规律。一个无量纲的中心化恒等式推翻了关于单纯形等角紧框架的系列假设。在14个不同规模的模型中，宏观类别结构仅承载4-12%的表示方差，而单个词元的上下文信息占79-91%，且在参数量相差100倍时保持稳定。理论层面，词元级权重衰减惩罚与类别类型数量成正比，而非出现频率，使下一个词预测退化为不平衡的K分类问题，最优解按类型数量排序类别范数。反向下界在二分类情况下被证明：类内离散度至少与条件互信息I(token; context | category)成正比。该定律成立：类内离散度（而非总方差）始终与信息量一致，无论模型或划分方式如何，且在模型间具有预测能力；预训练过程中，类别信息占比先超调、后下降并部分恢复，因所需携带的信息始终未消失。
原文摘要 · Abstract (English)
Within-class variance in language-model representations is commonly read as incomplete neural collapse. We argue it is allocated information storage, and that the allocation obeys a law. A one-line centering identity voids a family of simplex equiangular-tight-frame claims, including our own earlier ones; in dimensionless variance shares across 14 models, macro-category structure carries only 4-12% of representational variance and within-token context carries 79-91%, stable across a 100x parameter range. On the theory side, token-level weight decay penalizes a category in proportion to its type count, not its occurrence mass, reducing next-token prediction to an imbalanced K-class problem whose optimum orders category norms by type count. A converse floor, proved for binary categories, forces within-category dispersion to be at least proportional to the conditional mutual information I(token; context | category). The law holds: identity dispersion, not total variance, tracks this information across every tested model and partition, under a model-free estimate and even across models, where one model's information predicts another's dispersion; and over pretraining the category share overshoots, decays, and partially recovers, because the information it must carry never left.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。