通过分析模型各层表示动态,揭示了隐藏状态演化的深层规律。
Layer-wise Representation Dynamics: An Empirical Investigation Across Embedders and Base LLMs

- 提出三层测量框架,捕捉表示在子空间中的运动、邻近保持与最终层对齐
- 发现不同架构和任务下层间变化差异显著,且与下游性能强相关
- 可用于无标签模型选择和推理时层剪枝,其中GFMI表现最优
现代语言模型中隐藏状态在各层间变化显著,但多数层级分析仅关注单一维度。本文提出层间表示动态(LRD)框架,包含三个测量家族:基于Frenet的全局子空间运动(速度与曲率)、局部最近邻保留率(NRS),以及与最终层对齐的图滤波互信息(GFMI)。在31个模型(编码器与解码器类嵌入器及基础大模型)上应用于30个MTEB任务,揭示了仅从最终层无法察觉的架构与任务级差异。进一步用于两个应用:无标签模型选择和推理时层剪枝。结果显示,三类指标均与下游MTEB性能正相关,整体子空间位移(d_{0,L})最强;在小型基模型MMLU面板上方向一致。剪枝实验中,仅GFMI在15%与20%预算下优于随机剪枝,且在所有预算下中位变化最佳;Frenet仅在极轻预算有效,NRS则无法从选择迁移至剪枝。结果表明,层间结构为解释与部署决策提供有效信号。
原文摘要 · Abstract (English)
Hidden states change substantially across the layers of modern language models, but most layer-wise analyses focus on one aspect of that change. We propose Layer-wise Representation Dynamics (LRD), a framework with three layer-wise measurement families: Frenet (Grassmann speed and curvature) for global subspace motion, Neighborhood Retention Score (NRS) for local nearest-neighbor retention, and Graph Filtration Mutual Information (GFMI) for alignment with the final layer. Applying LRD to 31 models (encoder-based and decoder-based embedders, plus base LLMs) on 30 MTEB tasks reveals architectural and task-level differences that are not apparent from final-layer representations alone. We then use LRD for two applications: label-free model selection and inference-time layer pruning. For selection, all three model-level scores correlate positively with downstream MTEB performance, with end-to-end subspace displacement (d_{0,L}) the strongest, and the same direction holds on a smaller base-LLM MMLU panel. For pruning, GFMI is the only measurement-guided rule that beats Random at the 15% and 20% budgets and has the best median change at every budget. Frenet is effective only at the lightest budget, while NRS does not transfer from model selection to pruning. These results show that layer-wise structure provides signal for both interpretation and deployment decisions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。