提出语义参考系,让语言模型深层计算更稳定可读。
SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models
- 用固定锚点分离语义测量与残差动态,避免坐标漂移。
- 构建语义沃罗诺伊图,量化每层的语义归属与内部变化。
- 最小作用路径揭示低复杂度轨迹,关联参数效率与知识密度。
残差流分析关注语言模型计算随深度的变化,但中间解码需要跨层一致的读出坐标。若嵌入锚点与反嵌入读出在选段上不一致,看似的运动可能只是测量漂移而非真实计算。我们引入语义参考框架(SemRF),一种基于锚点的形式化方法,将语义测量与残差动态分离。一个SemRF固定锚点,并以之为基准测量状态。伪逆绑定实现精确同步;在受限双可逆条件下,SemRF产生稳定的语义基坐标、畸变界和近恒等变化。固定框架后,残差计算变为深度方向的语义轨迹。锚点诱导出语义沃罗诺伊图:距离或置信度(如logits)将每层分配至粗粒度单元,而坐标保留单元内运动与边界。我们定义逐层步长、贡献轮廓与失衡诊断,并利用沃罗诺伊轨迹定义松弛边界的管状区域。标准轨迹是该管内最小作用路径;当非空且带正二次权重时,其唯一且在活跃约束外满足离散样条方程。多余作用控制步长、曲率与轮廓偏差。低曲率意味着分段线性可压缩性与局部知识密度:轨迹复杂度越低,语义结点越少。通过参数到轨迹映射,这建立起与参数效率的条件关联:在适配数据的设定中,低作用与低复杂度轨迹使用更少的语义自由度。这些保证依赖于可控接口误差与显式管约束下的小投影残差。
原文摘要 · Abstract (English)
Residual-stream analysis asks how language-model computation evolves across depth, but intermediate decoding requires comparable readout coordinates across layers. If embedding anchors and unembedding readout disagree on the chosen span, apparent motion may reflect measurement drift rather than computation. We introduce \emph{Semantic Reference Frames} (SemRF), an anchor-based formalism separating semantic measurement from residual dynamics. A SemRF fixes anchors and measures states against them. Pseudo-inverse tying gives exact synchronization; under restricted bi-invertibility, SemRF yields stable semantic-basis coordinates, distortion bounds, and near-identity changes. With the frame fixed, residual computation becomes a depthwise semantic trajectory. The anchors induce a semantic Voronoi diagram: distance, or evidence such as logits, assigns each layer to a coarse cell, while coordinates retain within-cell motion and margins. We define layerwise steps, contribution profiles, and imbalance diagnostics, then use the Voronoi trace to define a margin-relaxed tube. The canonical trace is the minimum-action path inside this tube; when nonempty with positive quadratic weight, it is unique and obeys a discrete spline equation away from active constraints. Excess action controls step, curvature, and profile mismatch. Low curvature implies piecewise-linear compressibility and local knowledge density: lower trace complexity means fewer semantic knots. Through the parameter-to-trajectory map, this gives a conditional link to parameter efficiency: among admissible settings fitting data, lower-action and lower-complexity traces use fewer semantic degrees of freedom. The guarantees require controlled interface error and small projection residual under explicit tube constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。