揭示大模型词元表示流动的非线性本质与内在动力机制
The Depth Flow of Token Representations Is Nonlinear and Does Not Descend Its Own Density
- 用离散朗之万模型拟合词元轨迹，发现非线性漂移优于线性映射
- 漂移不沿自身对数密度下降，而是趋向一个非密度势能函数
- 旋转分量显著（4%~45%），保留词元角排名而打乱其范数与集中度排名
词元表示在神经网络中逐层传递，整个词汇表形成一种流动。本文基于Pythia-160M和Pythia-410M的语料库平均轨迹，将该流动建模为离散朗之万方程，并在保留词元上评估预测步数。线性映射常被用作廉价替代，但其总结的流动并非线性：二次漂移在两个模型的所有层间转移中均优于线性映射，且在局部邻域内克兰默-莫伊尔估计器结果一致。进一步分析表明，该流动并不沿自身对数密度下降，而是趋向一个非密度的势能函数；旋转分量不可忽略，占可解释漂移的4%至45%，且流动保持词元的角排名不变——尽管其范数排名被随机化，浓度排名在最后一层反转。
原文摘要 · Abstract (English)
A token's representation is carried through the network layer by layer. The whole vocabulary carried together forms a flow. We fit this flow's equation of motion as a discrete Langevin model over corpus-mean trajectories of Pythia-160M and Pythia-410M, and score the predicted steps on held-out tokens. Linear maps are often used as cheap surrogates for a layer. The flow they summarize is not linear: a quadratic drift beats the linear linear map at every transition of both models, and the Kramers--Moyal estimator agrees wherever its neighborhoods stay local. We then characterize the flow further. First, we show that it does not descend its own log-density. The drift instead descends a potential that is not the density. Second, the rotational component is not negligible, $4$ to $45\%$ of the explainable drift, and the circulation shows in what the flow preserves: a token keeps its angular rank across all thirteen layers while its norm rank is shuffled and its concentration rank is reversed by the last block.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。