中间层比末层更能捕捉丰富语义，提升下游任务表现。
Layer by Layer: Uncovering Hidden Representations in Language Models
- 提出基于信息论、几何与扰动不变性的统一评估框架
- 32项任务中中间层特征性能普遍优于末层
- 适用于各类模型架构，推动更鲁棒的表征学习
大型语言模型（LLMs）的输出通常依赖最终层，传统观点认为早期层仅捕获低级特征。但我们的分析表明，中间层可编码更丰富的表示，显著提升多种下游任务性能。为此，我们提出一个基于信息论、几何结构和输入扰动不变性的统一表示质量评估框架，揭示各层在信息压缩与信号保留之间的权衡机制。在32个文本嵌入任务上，涵盖变换器与状态空间模型等多种架构及语言、视觉等多领域，实验显示中间层嵌入持续优于末层，挑战了传统末层优先观念，为利用中深层表示构建更鲁棒、准确的表征开辟新路径。
原文摘要 · Abstract (English)
From extracting features to generating text, the outputs of large language models (LLMs) typically rely on the final layers, following the conventional wisdom that earlier layers capture only low-level cues. However, our analysis shows that intermediate layers can encode even richer representations, often improving performance on a range of downstream tasks. To explain and quantify these hidden-layer properties, we propose a unified framework of representation quality metrics based on information theory, geometry, and invariance to input perturbations. Our framework highlights how each layer balances information compression and signal preservation, revealing why mid-depth embeddings can exceed the last layer's performance. Through extensive experiments on 32 text-embedding tasks across various architectures (transformers, state-space models) and domains (language, vision), we demonstrate that intermediate layers consistently provide stronger features, challenging the standard view on final-layer embeddings and opening new directions on using mid-layer representations for more robust and accurate representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。